Medical device navigation using a virtual 3d space

ABSTRACT

A system and method for providing image guidance for placement of one or more medical devices at a target location. The system can determine one or more intersections between a medical device and an image region based at least in part on first emplacement data and second emplacement data. Using the determined intersections, the system can cause one or more displays to display perspective views of image guidance cues, including an intersection ghost in a virtual 3D space.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/415,398, filed on Jan. 25, 2017, which claims prioritybenefit to U.S. Provisional Application No. 62/413,925, each of which ishereby incorporated herein by reference in its entirety. Any and allapplications for which a foreign or domestic priority claim isidentified in the Application Data Sheet as filed with the presentapplication are incorporated by reference under 37 CFR 1.57 and made apart of this specification.

BACKGROUND

Various systems are available to aid a healthcare provider to guide amedical device in a patient or to provide a user viewing an object withadditional information. The systems can provide image guidance cues toaid the healthcare provider or user and can also provide additionalinformation for the user's benefit.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a diagram illustrating an embodiment of an environment forimage-guided medical procedures.

FIG. 1B is a diagram illustrating embodiments of coordinate systems thatcan be used by the system.

FIG. 2 is a diagram illustrating an embodiment of a rendering of imageguidance cues and medical display objects on a display.

FIG. 3 is a diagram illustrating an embodiment of a rendering of imageguidance cues and medical display objects, including a ghost medicaldevice.

FIG. 4 is a diagram illustrating an embodiment of a rendering of imageguidance cues and medical display objects, including a ghost medicalimage.

FIG. 5 is a diagram illustrating an embodiment of an affected regionassociated with a medical device.

FIG. 6 is a diagram illustrating an embodiment of a medical devicehaving visual tracking fiducials.

FIG. 7 is a diagram illustrating an embodiment of a rendering of a firstand second virtual medical device and a medical image.

FIG. 8 is a diagram illustrating an embodiment of a rendering first andsecond virtual medical device.

FIG. 9 is a diagram illustrating an embodiment of a medical devicehaving an integrated imaging device.

FIG. 10 is a diagram illustrating an embodiment of a medical devicehaving an integrated imaging device.

FIG. 11 is a diagram illustrating an embodiment of a medical devicehaving an integrated imaging device.

FIG. 12 is a diagram illustrating an embodiment of a medical devicehaving an integrated imaging device.

FIG. 13 is a flow diagram illustrative of an embodiment of a routineimplemented by the system to determine and cause a display to displayone or more intersection indicators corresponding to an intersectionbetween a plane-of-interest and one or more trajectories of a medicaldevice.

FIG. 14 is a flow diagram illustrative of an embodiment of a routineimplemented by the system to project a display object onto aplane-of-interest to display a ghost display object.

FIG. 15 is a flow diagram illustrative of an embodiment of a routineimplemented by the system to display an affected region of a medicaldevice.

FIG. 16 is a flow diagram illustrative of an embodiment of a routineimplemented by the system to determine emplacement of a virtual medicaldevice and a medical image based at least in part on receivedemplacement data associated with a medical device.

DETAILED DESCRIPTION

Implementations disclosed herein provide systems, methods, and apparatusfor displaying medical images, such as, but not limited to ultrasound,CT, and/or MRI images, facilitating medical device insertion into tissueby an operator. Certain embodiments pertain to a free-hand medicaldevice guidance system. The system can provide the healthcare providermanual control over the medical device, while making the spatialrelationships between the target, medical device and medical image (alsoreferred to as an image slice or rendered medical image), or image areacorresponding to the medical image (also referred to as an image slicearea or scan area), more intuitive via a visual display. Using thisvisual feedback, the operator can adjust the medical device's position,orientation, or trajectory. Certain of the contemplated embodiments canbe used in conjunction with systems described in greater detail in U.S.patent application Ser. No. 13/014,587, filed Jan. 26, 2011, entitledSYSTEMS, METHODS, APPARATUSES, AND COMPUTER-READABLE MEDIA FOR IMAGEMANAGEMENT IN IMAGE-GUIDED MEDICAL PROCEDURES and U.S. patentapplication Ser. No. 13/753,274, filed Jan. 29, 2013, entitled MULTIPLEMEDICAL DEVICE GUIDANCE (the '274 Application), U.S. patent applicationSer. No. 14/212,933, filed Mar. 14, 2014, entitled MEDICAL DEVICEGUIDANCE, U.S. patent application Ser. No. 15/199,630, filed Jun. 30,2016, entitled LOUPE DISPLAY, and U.S. patent application Ser. No.14/872,930, filed Oct. 1, 2015, entitled AFFECTED REGION DISPLAY (the'930 Application), each of which is hereby incorporated by reference inits entirety.

Medical interventions typically involve using an instrument to resect,cauterize, staple, seal, or otherwise manipulate soft tissue and organs.A physician must take great care to minimize blood loss and minimizedamage to ancillary tissue while performing these tissue-damaginginterventions. This is even more difficult with minimally-invasivesurgeries, such as laparoscopic, endoscopic, and robotic surgeries. Aphysician may use ultrasound to image the internal structures of anorgan before stapling, transecting, resecting, sealing, or graspingtissue, helping her avoid critical structures such as blood vessels.However, even with ultrasound imaging, there is a significantpossibility of inadvertent damage to surrounding tissue and bloodvessels during these procedures. This is because it is not obvious inthe externally displayed medical image where a given internal structureis located relative to the medical device.

The system can aid the healthcare provider in guiding one or moremedical devices through or around tissue of the patient and/or placingthe medical devices. The system can be used to aid in stapling,transecting, resecting, sealing, and/or grasping tissue. Additionally,the system can be used for treatment of tumors, fibroids, cysts, damagedblood vessels, or other damages internal structures of a patient. Thesystem can be used during open surgery, laparoscopic surgery, endoscopicprocedures, robotic surgeries, biopsies, and/or interventional radiologyprocedures.

The system can be used in conjunction with live intraoperativeultrasound (U/S), pre-operative CT, or any cross-sectional medicalimaging modality (e.g. MRI, OCT, etc.). In addition, the system can usea variety of techniques to determine the position and/or orientation ofone or more medical devices. For example, the system can use the NDIAurora magnetic system, NDI Polaris optical system, etc. In someembodiments, a position sensor can be embedded inside, or affixed toeach medical device, at the tip, along the shaft, and/or on the handle.Sensors can be built into the medical devices or attached aftermanufacturing, as described in greater detail in U.S. application Ser.No. 14/212,184, filed Mar. 14, 2014, entitled SENSOR MOUNT, incorporatedherein in its entirety.

Each medical device can be associated with one or more sensors, whichcan continually, or repeatedly, report position and/or orientation, or asingle sensor can be used for all the medical devices. In someembodiments where one sensor is used, the healthcare provider can attachthe sensor to the particular medical device that she is intentionallyrepositioning, and then, once she has placed that medical device, shecan remove the sensor and attach it to the next medical device she isrepositioning. In some embodiments, the medical devices can bemanipulated by the healthcare provider. In certain embodiments, thesystem can be used with a robotic manipulator, where the robot controlsthe medical devices. In some embodiments, visually-detectable fiducialscan be used to determine position and/or orientation for one or more ofthe medical devices.

In some embodiments, the handles of medical devices can have push-buttonswitches, to allow the user to select a medical device, indicate atissue target, etc. The handle can also have an indicator light toindicate to the users which medical device is selected. Finally, thehandle can have an encoder to detect how much length of electrode hasbeen exposed by the user, and report this information to the guidancesystem and therapeutic generator.

Image Guidance Systems

FIG. 1A is a diagram illustrating an embodiment of a system for imagemanagement in image-guided medical procedures. In some embodiments, theposition sensing unit 140 can track medical devices within a trackingarea and provide data to the image guidance unit 130. The medicaldevices can include invasive medical devices, such as, but not limitedto, graspers, staplers, vessel sealers, electrocautery devices,resecting devices, transecting devices, scalpels, biopsy needles,ablation needles, surgical needles, nerve-block needles, or otherneedles, catheters, stents, laparoscopes or laparoscopic cameras,ultrasound transducers, or other instruments that enter a part of thebody, and non-invasive medical devices that do not enter the body, suchas, but not limited to, ultrasound transducers, probes, or otherexternal imaging devices, etc. The medical devices can also includemedical imaging devices that provide or aid in the selection of medicalimages for display. In some embodiments, the medical imaging device canbe any device that is used to select a particular medical image fordisplay. The medical imaging devices can include invasive medicaldevices, such as laparoscopic cameras, and non-invasive medical devices,such as external ultrasound transducers.

Although only two medical devices 145 and 155 are shown in FIG. 1, itwill be understood that additional medical devices can be tracked andassociated data can be provided to the image guidance unit 130. Theimage guidance unit 130 can process or combine the data and show imageguidance data on display 120. This image guidance data can be used by ahealthcare provider to guide a procedure and improve care. There arenumerous other possible embodiments of system 100. For example, many ofthe depicted components can be joined together to form a singlecomponent and can be implemented in a single computer or machine.Further, additional position sensing units can be used in conjunctionwith position sensing unit 140 to track all relevant medical devices 145and 155, as discussed in more detail below. Additional imaging units 150can be included, and combined imaging data from the multiple imagingunits 150 can be processed by image guidance unit 130 and shown ondisplay unit 120. Additionally, two or more surgical systems 149 canalso be included.

Information about and from multiple surgical systems 149 and attachedmedical devices 145 (and additional medical devices not shown) can beprocessed by image guidance unit 130 and shown on display 120. These andother possible embodiments are discussed in more detail below. It willbe understood that any combination of the display objects (such as, butnot limited to medical display objects (e.g., virtual medical device,medical image, ghost display objects, etc.) and/or image guidance cues(e.g., trajectory indicators, intersection indicators, plane indicators,ghost display objects, etc.)) described herein can be displayedconcurrently, or simultaneously. Further, reference to displayingobjects “concurrently” and/or “simultaneously” is to be interpretedbroadly and may refer to displaying objects in such a way that to ahuman observer the objects are visible at the same time.

Imaging unit 150 can be coupled to image guidance unit 130. In someembodiments, imaging unit 150 can be coupled to a second display unit(not shown). The second display unit can display imaging data fromimaging unit 150. The imaging data displayed on display unit 120 anddisplayed on second display unit can be the same or different. In someembodiments, the imaging unit 150 is an ultrasound machine, the movableimaging device 155 is an ultrasound transducer or ultrasound probe, andthe second display unit is a display associated with the ultrasoundmachine 150 that displays the medical images from the ultrasound machine150. In some embodiments, a movable imaging unit 155 can be connected toimage guidance unit 130. The movable imaging unit 155 can be useful forallowing a user to indicate what portions of a first set of imaging dataare to be displayed. For example, the movable imaging unit 155 can be anultrasound transducer 155, a stapler or other medical device, forexample, and can be used by a user to indicate what portions of imagingdata, such as a pre-operative CT scan, to show on a display unit 120 asimage 125. Further, in some embodiments, there can be a third set ofpre-operative imaging data that can be displayed with the first set ofimaging data.

In some embodiments, system 100 comprises a display unit 120 and aposition sensing unit 140 communicatively coupled to image guidance unit130. In some embodiments, position sensing unit 140, display unit 120,and image guidance unit 130 are coupled to the stand 170. Image guidanceunit 130 can be used to produce images 125 that are displayed on displayunit 120. The images 125 produced on display unit 120 by the imageguidance unit 130 can be determined based on ultrasound or other visualimages from the first medical device 145 and second medical device 155.In the illustrated embodiment, the images 125 include a 2D viewing areaand a 3D viewing area. In the 2D viewing area, some or all of thedisplay objects can be displayed as 2D objects. For instance, the 2Dviewing area can include a 2D view of each of an ultrasound slice, avirtual medical device corresponding to the first medical device, avirtual imaging device corresponding to the second medical device,intersection indicator, trajectory indicators and/or other imageguidance cues. It will be understood that some or all of the displayobjects in the 2D viewing area can be displayed as 3D objects. In theillustrated embodiment, the 3D viewing area includes perspective viewsof each of the medical image 121, the virtual medical device 122, adisplayed affected region 129, the virtual imaging device 123,intersection indicator 126, trajectory and other image guidance cues127A, 127B (generically referred as image guidance cues 127 and/ortrajectory indicators 127), and a patient orientation indicator 128. Itwill be understood that any combination of the aforementioned displayobjects can be displayed in the 2D view and/or 3D view as desired.

As a non-limiting example, if the first medical device 145 is a stapler145 having a joint member and two limb members coupled to the jointmember at different angles and the second medical device 155 is anultrasound probe 155, then images 125 produced on display 120 caninclude the images, or video, from the ultrasound probe 155 (e.g.,medical image 121) combined with other medical display objects and imageguidance cues, such as display object trajectories (e.g., trajectoryindicators 127) or affected region 129 (e.g., trajectory of areacorresponding to a volume located between the first limb member and thesecond limb member), determined based on the emplacement of the stapler145. If the first medical device 145 is an ultrasound probe 145 and thesecond medical device 155 is a laparoscopic camera 155, then images 125produced on display 120 can include the video from the laparoscopiccamera 155 combined with ultrasound data superimposed on thelaparoscopic image. More medical devices can be added to the system. Forexample, the system can include a stapler, ultrasound probe, ablationneedle, laparoscopic camera, scalpel, grasper, vessel sealer,electrocauterizer, resecting device, transecting device and/or any othersurgical instrument or medical device. The system can also processand/or display collected data, such as preoperative CT scans, X-Rays,MRIs, laser scanned 3D surfaces etc.

The term “emplacement” as used herein is a broad term and may refer to,without limitation, position and/or orientation or any other appropriatelocation information. The term “pose” as used herein is a broad termencompassing its plain and ordinary meaning and may refer to, withoutlimitation, position and orientation or any other appropriate locationinformation. In some embodiments, the imaging data obtained from one orboth of medical devices 145 and 155 can include other modalities such asa CT scan, MRI, open-magnet MRI, optical coherence tomography (“OCT”),positron emission tomography (“PET”) scans, fluoroscopy, ultrasound, orother preoperative, or intraoperative 2D or 3D anatomical imaging data.In some embodiments, medical devices 145 and 155 can also be scalpels,implantable hardware, or any other device used in surgery. Anyappropriate surgical system 149 or imaging unit 150 can be attached tothe corresponding medical instruments 145 and 155.

As noted above, images 125 produced can also be generated based on live,intraoperative, or real-time data obtained using the second medicaldevice 155, which is coupled to second imaging unit 150. The term “realtime” as used herein is a broad term and has its ordinary and customarymeaning, including without limitation instantaneously or nearlyinstantaneously. The use of the term real time can also mean thatactions are performed or data is obtained with the intention to be usedimmediately, upon the next cycle of a system or control loop, or anyother appropriate meaning. Additionally, as used herein, real-time datacan be data that is obtained at a frequency that would allow ahealthcare provider to meaningfully interact with the data duringsurgery. For example, in some embodiments, real-time data can be amedical image of a patient that is updated one time per second. In someembodiments, real-time data can be ultrasound data that is updatedmultiple times per second.

The medical devices 145, 155 can be communicatively coupled to theposition sensing unit 140 (e.g., sensors embedded or coupled to themedical devices 145, 155 can be communicatively coupled with theposition sensing unit 140). The position sensing unit 140 can be part ofimaging unit 150 or it can be separate. The position sensing unit 140can be used to determine the emplacement of first medical device 145and/or the second medical device 155. In some embodiments, the positionsensing unit 140 can include a magnetic tracker and/or one or moremagnetic coils can be coupled to medical devices 145 and/or 155. In someembodiments, the position sensing unit 140 can include an opticaltracker and/or one or more visually-detectable fiducials can be coupledto medical devices 145 and/or 155. In some embodiments, the positionsensing unit 140 can be located below the patient. In such embodiments,the position sensing unit 140 can be located on or below the table 180.For example, in embodiments where the position sensing unit 140 is amagnetic tracker, it can be mounted below the surgical table 180. Suchan arrangement can be useful when the tracking volume of the positionsensing unit 140 is dependent on the location of the position sensingunit 140, as with many magnetic trackers. In some embodiments, magnetictracking coils can be mounted in or on the medical devices 145 and 155.

In some embodiments, the position sensing unit 140 can be anelectromagnetic measurement system (e.g., NDI Aurora system) usingsensor coils for tracking units attached to the first and/or secondsurgical devices 145 and 155. In some embodiments, the second positionsensing unit 140 can be an optical 3D tracking system using fiducials.Such optical 3D tracking systems can include the NDI Polaris Spectra,Vicra, Certus, PhaseSpace IMPULSE, Vicon MX, InterSense IS-900,NaturalPoint OptiTrack, Polhemus FastTrak, IsoTrak, or ClaronMicronTracker2. In some embodiments, the position sensing unit 140 caneach be an inertial 3D tracking system comprising a compass,accelerometer, tilt sensor, and/or gyro, such as the InterSenseInertiaCube or the Nintendo Wii controller. In some embodiments, theposition sensing unit 140 can be attached to or affixed on thecorresponding surgical device 145 and 155.

In some embodiments, the position sensing units 140, can include sensingdevices such as the HiBall tracking system, a GPS device, or signalemitting device that would allow for tracking of the position and/ororientation (e.g., emplacement) of the tracking unit (also referred toas an emplacement sensor). In some embodiments, a position sensing unit140 can be affixed to either or both of the surgical devices 145 and155. The surgical devices 145 or 155 can be tracked by the positionsensing unit 140. A room coordinate system reference, such as thedisplay 120 can also be tracked by the position sensing unit 140 inorder to determine the emplacements of the surgical devices 145 and 155with respect to the room coordinate system. Devices 145 and 155 can alsoinclude or have coupled thereto one or more accelerometers, which can beused to estimate movement, position, and location of the devices.

In some embodiments, the position sensing unit 140 can be an AscensionFlock of Birds, Nest of Birds, driveBAY, medSAFE, trakSTAR, miniBIRD,MotionSTAR, pciBIRD, or Calypso 2D Localization System and trackingunits attached to the first and/or second medical devices 145 and 155can be magnetic tracking coils.

The term “tracking unit” (also referred to as an emplacement sensor), asused herein, is a broad term encompassing its plain and ordinary meaningand includes without limitation all types of magnetic coils or othermagnetic field sensing devices for use with magnetic trackers, fiducialsor other optically detectable markers for use with optical trackers,such as those discussed above and below. In some embodiments, thetracking units can be implemented using optical position sensingdevices, such as the HiBall tracking system and the position sensingunit 140 can form part of the HiB all tracking system. Tracking unitscan also include a GPS device or signal emitting device that allows fortracking of the position and/or orientation of the tracking unit. Insome embodiments, a signal emitting device might include aradio-frequency identifier (RFID). In such embodiments, the positionsensing unit 140 can use the GPS coordinates of the tracking units orcan, for example, triangulate the radio frequency signal being emittedby the RFID associated with tracking units. The tracking systems canalso include one or more 3D mice.

Images 125 can be produced based on intraoperative or real-time dataobtained using first medical device 145, which is coupled to firstsurgical system 149. In the illustrated embodiment of FIG. 1, the firstsurgical system 149 is shown as coupled to image guidance unit 130. Thecoupling between the first surgical system 149 and image guidance unit130 may not be present in all embodiments. In some embodiments, thecoupling between first surgical system 149 and image guidance unit 130can be included where information about first medical device 145available to first surgical system 149 is useful for the processingperformed by image guidance unit 130. For example, in some embodiments,the first medical device 145 (for example, a stapler) includes a jointmember and two limb members coupled to the joint member at differentangles and the first surgical system 149 is a stapler navigation system149. In some embodiments, it can be useful to send a signal about theangle of one or more of the jaws of instruments such as staplers, vesselsealers, and graspers to image guidance unit 130 so that image guidanceunit 130 can show an affected region of tissue which is located betweenthe two jaws. In other embodiments, the first surgical system 149 is notcoupled to image guidance unit 130. Example embodiments including imagesand graphics that can be displayed are included below.

In some embodiments, the display unit 120 displays 3D images to a user,such as a healthcare provider. Stereoscopic 3D displays separate theimagery shown to each of the user's eyes. This can be accomplished by astereoscopic display, a lenticular auto-stereoscopic display, or anyother appropriate type of display. The display 120 can be an alternatingrow or alternating column display. Example alternating row displaysinclude the Miracube G240S, as well as Zalman Trimon Monitors.Alternating column displays include devices manufactured by Sharp, aswell as many “auto-stereoscopic” displays (e.g., Philips). In someembodiments, Sony Panasonic 3D passive displays and LG, Samsung, and/orVizio 3D TVs can be used as well. Display 120 can also be a cathode raytube. Cathode Ray Tube (CRT) based devices, can use temporal sequencing,showing imagery for the left and right eye in temporal sequentialalternation. This method can also be used for projection-based devices,as well as by liquid crystal display (LCD) devices, light emitting diode(LED) devices, and/or organic LED (OLED) devices.

In certain embodiments, the display unit 120 can be a head mounteddisplay worn by the user in order to receive 3D images from the imageguidance unit 130. In such embodiments, a separate display, such as thepictured display unit 120, can be omitted. The 3D graphics can beproduced using underlying data models, stored in the image guidance unit130 and projected onto one or more 2D planes in order to create left andright eye images for a head mount, lenticular, or other 3D display. Theunderlying 3D model can be updated based on the relative emplacements ofthe various devices 145 and 155, as determined by the position sensingunit(s) 140, and/or based on new data associated with the devices 145and 155. For example, if the second medical device 155 is an ultrasoundprobe, then the underlying data model can be updated to reflect the mostrecent medical image. If the first medical device 145 is a stapler, thenthe underlying model can be updated to reflect any changes related tothe jaws, such as information regarding the affected region or angles ofthe jaws and/or transecting knife. Any appropriate 3D graphicsprocessing can be used for rendering including processing based onOpenGL, Direct3D, Java 3D, etc. Whole, partial, or modified 3D graphicspackages can also be used, such packages including 3DS Max, SolidWorks,Maya, Form Z, Cybermotion 3D, VTK, Slicer, or any others. In someembodiments, various parts of the needed rendering can occur ontraditional or specialized graphics hardware. The rendering can alsooccur on the general CPU, on programmable hardware, on a separateprocessor, be distributed over multiple processors, over multiplededicated graphics cards, or using any other appropriate combination ofhardware or technique.

One or more components, units, devices, or elements of variousembodiments can be packaged and/or distributed as part of a kit. Forexample, in one embodiment, a stapler, one or more tracking units, 3Dviewing glasses, and/or a portion of an ultrasound wand can form a kit.Other embodiments can have different elements or combinations ofelements grouped and/or packaged together. Kits can be sold ordistributed separately from or with the other portions of the system.

One will readily recognize that there are numerous other examples ofimage guidance systems which can use, incorporate, support, or providefor the techniques, methods, processes, and systems described herein.

Coordinate Systems

FIG. 1B is a diagram illustrating embodiments of coordinate systems thatcan be used by the system 100. The system 100 can utilize the coordinatesystems to track and display the various display objects on thedisplays, including, but not limited to one or more position sensingcoordinate systems 180A, 180B (generically referred to as 180), one ormore room coordinate systems 182, one or more 3D scene coordinatesystems 184, one or more display coordinate systems 186, one or moresensor coordinate systems 188A, 188B, one or more medical devicecoordinate systems 190A, 190B, 190C, 190D, one or more virtual medicaldevice coordinate systems 192A, 192B, 192C, 192D, one or more medicalimage coordinate systems 194, etc.

A position sensing coordinate system 180 can be used to determine theemplacement of various objects within a position sensing region. Forexample, in some embodiments, the position sensing coordinate system 180can refer to the coordinate system 180A used by a magnetic tracker (notshown) tracking objects within a magnetic field volume, or to thecoordinate system 180B used by an optical tracker 196 tracking objectswithin a volume. In some cases, multiple position sensing coordinatesystems 180 can be used together. For example, a magnetic positionsensing coordinate system 180A can be used in conjunction with amagnetic tracker tracking sensor coils within a position sensing regionthat are coupled to medical devices and an optical position sensingcoordinate system 180B can be used in conjunction with an opticaltracker tracking a fiducial coupled to a head mounted display (HMD) or auser, or to an optical tracker analyzing an image captured by an imagesensor. It will be understood that any combination of the trackersystems and position sensing coordinate systems 180 can be used asdesired. For example, in some cases, a single position sensingcoordinate system 180 can be used to track tracking sensors associatedwith one or more medical devices and/or tracking sensors associated witha user or HMD. Similarly, a different position sensing coordinate system180 can be used for each tracking sensor, or any combination thereof.

Room coordinate systems 182 can be used to determine the emplacement ofobjects within a room, such as an operating room. For example, the roomcoordinate system 182 can be used to determine or identify the relativeemplacement of the position sensing unit, medical devices, trackingsensors, user, display, etc. relative to each other within a room.

A 3D scene coordinate system 184, which may also be referred to as a 3Dvolume or scene graph coordinate system, can be used to determine theemplacement of display objects within a virtual 3D scene. In some cases,the 3D scene coordinate system 184 can identify the relative emplacementof virtual objects within the 3D scene. In certain embodiments, thevirtual objects can correspond to real objects, such as to medicaldevices 145, 155 and/or to computer-generated objects, such as such astrajectory cues 127, affected region 129, etc. Additionally, in certainembodiments, display objects can correspond to real objects, virtualobjects, and/or computer generated objects.

A display coordinate system 186 can be used to determine the emplacementof display objects for display on the display 102. For example, thedisplay coordinate system 186 can be used to determine the emplacementof virtual medical devices, medical images, image guidance cues, and thelike, within a display 102. In some embodiments, the display coordinatesystem 186 can be used to determine how the objects within the 3D sceneare to be displayed on the display. For example, the display coordinatesystem 186 can be used to determine the point-of-view location, or eyepoint, relative to the 3D scene (or 3D volume coordinate system 184) orscene graph for viewing the contents of the 3D scene. As mentionedabove, multiple display coordinate systems 186 can be used. For example,left-eye, right-eye, and/or center-eye display coordinate systems 186can be used to display different perspective of the display objectswithin a 3D scene, such as when a 3D display and/or an HMD is beingused.

A medical image coordinate system 194 can be used in conjunction withmedical images used and/or processed by the system. As describedpreviously, the medical images can be ultrasound images, CT image, MRI,images, etc. The images can be different sizes or shapes. For example,one ultrasound can output an image having one size and shape while adifferent ultrasound can output an image having a different size and/orshape. Similarly, CT, MRI, and ultrasound images may have differentsizes and shapes. Accordingly, the medical image coordinate system canbe used to identify the particular size and shape of the medical imagebeing used and/or processed by the system 100.

It will be understood that fewer, more, or different coordinate systemscan be used as desired. For example, in some embodiments, the 3D scenecoordinate system 184 can be omitted or combined with display coordinatesystem 186 and/or the position sensing coordinate system 180.Furthermore, in some cases, one or more tracking sensor coordinatesystems 188A, 188B, medical device coordinate systems 190A, 190B, 190C,190D, virtual medical device coordinate systems 192A, 192B, 192C, 192D,or other objects etc., can have their own coordinate system. Thecoordinate systems for the tracking sensors, medical devices, and/orvirtual medical devices can be used to identify the dimensions of thesensor/device/display object and relationship of thesensor/device/display object to another sensor/device/display object orother coordinate systems. For example, a medical device coordinatesystem (or virtual medical device coordinate system) can identify thedimensions of a corresponding medical device or virtual medical device,as well as the emplacement of a tracking sensor relative to the medicaldevice (or vice versa). Similarly, a medical imaging device coordinatesystem can identify the dimensions of the corresponding medical imagingdevice (or virtual medical imaging device) and/or an emplacement of amedical image relative to the medical imaging device (non-limitingexample: the emplacement of an ultrasound image relative to thecorresponding ultrasound transducer), or vice versa. The system 100 canuse various coordinate systems to determine the emplacement of a portionor the entire object with respect to each other and with respect to theother coordinate systems.

The system 100 can use the various coordinate systems to determineemplacement of objects relative to each other and determine how todisplay the display objects on a display, such as the display 102. As anon-limiting example, to display a virtual rendering of an ultrasoundtransducer 123 and ultrasound image 121 on the display 102, the system100 can determine the emplacement of a magnetic tracking sensor coupledto the ultrasound transducer 155 within a magnetic position sensingcoordinate system 180. Using a magnetic tracking sensor coordinatesystem 188B, the system 100 can determine the location of each portionof the magnetic tracking sensor within the magnetic position sensingcoordinate system 180. The system 100 can also determine the emplacementof each portion the ultrasound transducer 155 within the magneticposition sensing coordinate system by mapping the ultrasound transducercoordinate system 190B to the magnetic tracking sensor coordinate system188B (or vice versa) and/or to the magnetic position sensing coordinatesystem 180.

In addition, the system 100 can map each portion of the ultrasound image121 corresponding to the ultrasound transducer 155 to the magneticposition sensing coordinate system 180 by mapping an ultrasound imagecoordinate system 194 to the ultrasound transducer coordinate system190B, the magnetic tracking sensor coordinate system 188B, and/or to themagnetic position sensing coordinate system 180.

To display the virtual ultrasound transducer 123 and ultrasound imageslice 121, the system 100 can map the various objects from the magneticposition sensing coordinate system 180 to a room coordinate system 182,which can identify the relative emplacement of the position sensingcoordinate system 180 to a display. The system can then map data to the3D scene coordinate system 194 and/or the display coordinate system 186.For 3D viewing, the system 100 can map the objects to multiple displaycoordinate systems 186, such as left-eye and/or right-eye coordinatesystems.

With continued reference to the non-limiting example, the system 100 candetermine an emplacement of an optical tracking sensor corresponding toa user within an optical position sensing coordinate system 180B. Theemplacement of the optical tracking sensor within the optical positionsensing coordinate system 180B can be mapped to the room coordinatesystem 182, the 3D scene coordinate system 184, and/or the displaycoordinate systems 186 for display. In this way the system 100 candetermine the emplacement of the ultrasound transducer 155 andultrasound image slice 121 relative to the user and display a virtualrendering of the ultrasound transducer 123 and ultrasound image slice121 within the 3D scene relative to the determined emplacement of theuser.

To display the virtual ultrasound transducer 123 and ultrasound imageslice 121, the system 100 can map the various objects from the magneticposition sensing coordinate system 180 to a room coordinate system 182,which can identify the relative emplacement of the position sensingcoordinate system 180 to a display. The system can then map data to the3D scene coordinate system 194 and/or the display coordinate system 186.For 3D viewing, the system 100 can map the objects to multiple displaycoordinate systems 186, such as left-eye and/or right-eye coordinatesystems.

Although the non-limiting examples have been described as mapping thevarious objects and coordinate systems, to a position sensing coordinatesystem 180, the room coordinate system 182, the 3D scene coordinatesystem 184, and to display coordinate systems 186, it will be understoodthat the one or more of the objects or coordinate systems can be mappeddirectly or indirectly to any other coordinate system. For example, themedical device image can be mapped directly to a left-eye displaycoordinate system 186, etc. Thus, any of the real or virtual objectsdescribed herein may be represented, detected, or imaged in anycoordinate system, and conversion between the various coordinate systemscan be performed in components of the system such as image guidance unit130, position sensing unit 140, imager 150, the an HMD, or othercomponents.

Furthermore, it will be understood that once the system 100 determinesan emplacement of a medical device in one coordinate system, such as aposition sensing coordinate system 180, the system 100 can determine theemplacement of a corresponding virtual medical device in a differentcoordinate system, such as the 3D scene coordinate system 184 or thescreen coordinate system 186, by mapping the coordinates of the firstcoordinate system to the coordinates of the second coordinate system, orvice versa. Accordingly, references made herein to determining anemplacement of the medical device can also refer to determining anemplacement of a virtual medical device corresponding to the medicaldevice, or vice versa. Similarly, references made herein to determiningan emplacement of a display object (non-limiting example: medical image)relative to the medical device can also refer to determining theemplacement of the display object relative to a corresponding virtualmedical device.

Depicting Medical Devices

It can often be difficult to discern the content of a 3D scene from a 2Ddepiction of it, or even from a 3D depiction of it. Therefore, variousembodiments herein provide image guidance that can help the healthcareprovider better understand the scene, relative emplacements or poses ofobject in the scene and thereby provide improved image guidance.

FIG. 2 illustrates a perspective view of a virtual rendering 202 of amedical device 242 being displayed on a screen 220 with a perspectiveview of a medical image 204. In some embodiments, the screen 220 cancorrespond to the screen of a display unit 120, which can be implementedusing a TV, computer screen, head-mounted display, projector, etc. Inthe illustrated embodiment, the virtual medical device 202 displayed onthe screen 220 corresponds to the stapler 242. A wire 246 connecting thestapler 242 to a stapler navigation system is also depicted in FIG. 2.

Although only one virtual medical device 202 is displayed, it will beunderstood that multiple medical devices can be tracked and displayedconcurrently, or simultaneously, on screen 220, as described in greaterdetail in the '274 Application, previously incorporated by reference.For example, a virtual rendering of the medical imaging device 222 canbe displayed.

The virtual medical device 202 (which can also be referred to as anavatar) can be displayed in a virtual 3D space with the screen 220acting as a window into the virtual 3D space. Thus, as the medicaldevice 242 is moved to the right with respect to a point-of-viewlocation (e.g., the location of the point-of-view for viewing the 3Dspace), the virtual medical device 202 also moves to the right.Similarly, if the medical device 242 is rotated 90 degrees so that thetip of the medical device is pointing away from the point-of-viewlocation (e.g., at the screen 220), the virtual medical device 202 willlikewise show the change in orientation, and show the tip of the virtualmedical device 202 in the background and the other end of the virtualmedical device 202 in the foreground. In some embodiments, as describedin greater detail in U.S. patent application Ser. No. 14/212,933, filedMar. 14, 2014, entitled MEDICAL DEVICE GUIDANCE (the '933 Application),incorporated herein by reference in its entirety, the point-of-viewlocation can be a fixed location, such as a predetermined distance/anglefrom the screen 220 or stand 170 and/or a location configured by theuser; or the point-of-view location can by dynamic. For example, thesystem can track a user in real-time and determine the point-of-viewlocation based at least in part on the tracked location of the user.

Some models of medical devices have markings such as bands around theshaft or jaws to indicate distance along the shaft or jaws. Healthcareproviders performing medical device procedures are often familiar withthese markings and can use them to help understand the spatialrelationship between the medical device and anatomy. In someembodiments, the make and model of the medical device 242 is known tothe image guidance system and the virtual medical device 202 displayedin display 220 can resemble medical device 242. The features of medicaldevices that can be rendered in the scene include the overall shape(diameter, angles, cross sectional shape, curvature, etc.), color,distance markers, angle of the jaws, visuals or echogenic fiduciarymarkers, the state of deployable elements such as tines, paddles,anchors, resection loops, stiffening or steerable sleeves, temperature,radiation, light or magnetic field sensors, lens, waveguides, fluidtransfer channels, and the like.

The type of medical device being used can be input into the imageguidance system 100, can be a system default, can be detected by acamera or other device, can be received as data from an attached medicaldevice, such as surgical system 149 in FIG. 1, or the information can bereceived in any other appropriate manner. Displaying on display 220, avirtual medical device 202 that resembles the medical device 242 canhelp healthcare providers associate the image guidance data with thereal world and can provide more familiar guidance information to ahealthcare provider, thereby further aiding the healthcare provider inthe guidance task. For example, the healthcare provider can see thefamiliar markings on the medical device 202 being displayed on thedisplay 220 and therefore be familiar with the distance and relativeplacement of the displayed medical device 202 with respect to otherdata, such as tissue 212 seen in a medical image 204, 205. Thisknowledge of relative placement of items being displayed can help thehealthcare provider move the medical device 242 into place. It should beunderstood that the terms medical device and virtual medical device cansometimes be used interchangeably as they can generally relate to thesame object, the medical device being the object in the real world andvirtual medical device being a representation of the object, such as anavatar, in virtual space. Additionally, in some instances virtualmedical device can sometimes be referred to as virtual surgicalinstrument, surgical instrument, rendered surgical instrument, renderedmedical device, avatar, rendered avatar, and virtual avatar.

Consider an embodiment in which the virtual medical device 202 in thedisplay 220 is a virtual stapler depicting the portion of the staplerthat will perform the stapling. The displayed virtual medical device caninclude a joint member 211, a first limb member 213, and a second limbmember 210. If the display 220 also includes ultrasound data, then thedoctor can find the tissue she wishes to staple by moving the ultrasoundprobe 222 until she identifies the target tissue. In variousembodiments, she will be able to see the displayed ultrasound data andits location relative to the displayed medical device 202. She can thendirect the medical device 242 until she sees, on display 220, that theaffected region of the virtual medical device 202 encompasses a desiredstapled region of the tissue in the medical image. When she activatesthe stapler 242, she can have a higher degree of confidence that she hasstapled the correct portion of the tissue. Various embodiments of thisare discussed below.

As another example, consider the physical markings that can be on theinstruments themselves. These markings can help orient a healthcareprovider during use of the instrument. In some embodiments, the imageguidance unit can represent these markings in the images displayed inthe display. For example, certain ultrasound transducers are built withan orientation mark (e.g., a small bump) on one side of the transducingarray. That mark can also be shown in the medical image on the scanner'sdisplay, to help the healthcare provider understand where the scannedanatomical structures shown on screen are located under the transducer,inside the patient.

In some embodiments, the image guidance system can display a symbolic 3Drepresentation of an orientation mark 214 both next to themotion-tracked ultrasound slice (e.g., moving with the displayedultrasound slice) and next to the 2D view of the ultrasound slice alsodisplayed by the system. An example of this orientation mark isdisplayed in FIG. 2, where a small rectilinear volume 214 is shown bothin proximity to the ultrasound slice displayed in the 3D view and theultrasound slice displayed in a 2D view. In some embodiments, theorientation mark 214 corresponds to a feature, such as a physicalmarking, of the ultrasound probe. In some embodiments, the orientationmark 214 is displayed to provide assistance in associating the 3D viewand the 2D view.

It will be understood that a medical image can correspond to image datareceived from an imaging device, such as an ultrasound transducer. Insome embodiments, the image data can correspond to a cross-section oftissue having a certain thickness. In some instances, the imaging devicecan compact the image data, and/or treat the image data as 2D data, suchthat there is no perceived thickness. In certain embodiments, when themedical image is displayed in a 3D view, the system can treat themedical image as a 2D or quasi 2D object. In such embodiments, thesystem can cause the medical image to have little to no perceptiblethickness. Accordingly, in certain embodiments, when the medical imageis oriented orthogonally or perpendicularly with respect to thepoint-of-view location, the system can cause the display to displaynothing or a line having a relatively small thickness, such as a fewpixels, etc. In some cases, the number of pixels used to display therelatively small thickness of the medical image can correspond to thesize of the display. For example, more pixels can be used for a largerdisplay and fewer pixels can be used for a smaller display, etc.

Other embodiments can track and display other types of instruments andtheir features. For example, a healthcare provider may want to track oneor more of a stapler, vessel sealer, grasper, scalpel, a biopsy, acauterizer (including an electrocauterizer and Bovies), forceps, cuttingloops on hysteroscopes, harmonic sheers, lasers (including CO₂ lasers),etc. For example, in various embodiments, the following devices can betracked and various aspects of their design displayed on display 220:Olympus™ OES Pro Hystero-Resectoscope, SonoSurg Ultrasonic SurgicalSystem Olympus™ GF-UC 160 Endoscope Wallus™ Embryo Transfer CatheterAngioDynamics® NanoKnife™, VenaCure™ laser, StarBurst, Uniblade, Habib®Resector Bovie™ Electrodes, Covidien Evident™, Cool-tip™ AblationAntennas, Opti4™ Electrodes Microsulis MEA (microwave endometrialablation), Acculis Halt™ Medical System Optimed BigLumen AspirationCatheter Optimed Optipure Stent Central venous catheterizationintroducer medical device (such as those made by Bard and Arrow).

Once tracked, a healthcare provider is able to see image guidance dataon display 220 that will allow her to know the relative pose, location,or emplacement of the tracked instrument(s) with respect to one anotheror with respect to imaging data and will be able to see, on display 220,the features of the instrument rendered in the scene.

With continued reference to FIG. 2, in some embodiments, the imageguidance system can constantly display an additional 2D view 205 of themedical image, simultaneous to the 3D depiction 204, so that the medicalimage is always visible, regardless of the emplacement in which thehealthcare provider holds the transducer 222. The 2D view 205 of themedical data can be similar to what a healthcare provider is accustomedto seeing with traditional medical displays, such as ultrasounddisplays. This can be useful to provide the healthcare provider withimaging to which she is accustomed and allows a healthcare provider tosee the medical data regardless of the then-current emplacement of theimaging device with respect to the user.

In some embodiments, the 2D view 205 of the medical image is depicted inthe upper right corner of the monitor (though it can be placed in anylocation). In some embodiments, the guidance system can automatically(and continually) choose a corner in which to render the 2D view 205 ofthe medical image, based on the 3D position of the medical devices inthe rendered scene. For example, in FIG. 2, stapler 242 can be held inthe healthcare provider's left hand and the medical device shaft is tothe left of the 3D view of the medical image, so that the 2D view 202 ofthe medical image in the upper right corner of display 220 does notcover any of the 3D features of the medical device (or vice-versa). Ifthe medical device were held in the healthcare provider's right hand,the virtual medical device shaft would appear on the right side. Toprevent the 2D view 205 in the corner of display 220 from covering themedical device shaft, the system can automatically move it to a cornerthat would not otherwise be occupied by graphics or data.

In some embodiments, the system attempts to avoid having the 2D view 205of the medical image quickly moving among corners of the display inorder to avoid overlapping with graphics and data in the display. Forexample, a function f can be used to determine which corner is mostsuitable for the 2D medical image to be drawn in. The inputs to f caninclude the locations, in the screen coordinate system, of the displayedmedical device tip, the corners of the 3D view of the medical image,etc. In some embodiments, f s output for any given point in time isindependent of f s output in the previous frames, which can cause themedical image to move among corners of the display rapidly. In someembodiments, the image guidance system will filter f s output over time.For example, the output of a filter g, for any given frame, could be thecorner, which has been output by f the most number of times over thelast n frames, possibly weighting the most recent values for f mostheavily. The output of the filter g can be used to determine in whichcorner of display 220 to display the 2D medical image and the temporalfiltering provided by g can allow the 2D view 205 of the medical imagedisplay to move more smoothly among the corners of the display 220.

In some embodiments, other appropriate virtual information and/or imageguidance cues can be overlaid on the 2D view 205 of the medical image aswell as the 3D view 204. Examples include: orientation indicator 214, anindication of the distance between the medical device's tip (forexample, the tip of the first limb member or the tip of the second limbmember) and the point in the plane of the medical image that is closestto the medical device tip; ghost affected area (e.g., a cross section oroutline of the affected region projected on the ultrasound slice);and/or intersection indicators (e.g., a point, box, outline, etc.)indicating an intersection between one or more axes or trajectories of adisplay object and a plane-of-interest (e.g., a medical image plane, amedical device plane, etc.).

Furthermore, it will be understood that other image guidance cues can begenerated and displayed on the display as described in greater detail inthe '274 Application, previously incorporated herein by reference. Forexample, the system 100 can generate and/or display graphical indicatorsthat help indicate the spatial relationship between a medical device andan medical image plane (e.g., graphical image plane indicators) or otherplane (e.g., graphical plane indicators), indicators to indicate therelative positions of the medical device(s) and medical image(s),features of interest, annotations, plane indicators, plane intersectionindicators, other graphical indicators, approximate medical devicelocation indicators, etc. As described in greater detail above and inthe '274 Application, the various image guidance cues can be generatedbased at least in part on the emplacement information of the medicaldevices used with the system 100.

Multi-Axis Device

With reference to FIGS. 1B and 2, one or more of the medical devices orthe system 100 can be a multi-axis medical device. For instance, themulti-axis medical device 203 can be a number of devices including twoor more axes or jaws, such as, but not limited to graspers, staplers,vessel sealers, electrocauterizers, resecting devices, transectingdevices and scalpels. As depicted, a distal end of the multi-axismedical device 202 includes a joint member 211, and two limb members213, 210 coupled to the joint member 211 at different angles. In someexamples, the multi-axis medical device 202 can have a number ofassociated axes. For instance, the joint member 211 can have a jointmember axis 198A, the first limb member 213 can have a first axis 198B,and the second limb member 210 can have a second axis 198C that isdifferent from the joint member axis 198A and the first axis 198B. Insome embodiments, such as the illustrated embodiment of FIG. 2, thejoint member axis 198A is the same as the first axis 198B. However, itwill be understood that in some cases, the joint member axis 198A, firstaxis 198B, and second axis 198C can all be different. For example, thefirst member 213 may be angled relative to the joint member 211 and thesecond member 210.

In some examples, the multi-axis medical device 202 can have one or moreprimary axis and/or one or more secondary axis. For instance, in someembodiments, the primary axis can correspond to the longitudinal axis ofa body of a medical device or portion of a medical device where two ormore limbs are joined, such as joint member 211 (or joint member axis198A) and the secondary axis or axes can correspond to the longitudinalaxis or axes of the limb members of the medical device, such as limbmembers 213, 210 (or first axis 198B and/or second axis 198C). In somecases, one or more of the secondary axes can be coincident with theprimary axis or parallel to the primary axis. In certain cases, one ormore of the secondary axes can move relative to the primary axis.

The one or more limb members 213, 210 can be movable and/or fixed. Forinstance, in the illustrated embodiment of FIG. 2, the first limb member213 (or lower jaw) is fixed relative to the joint member 211, and thesecondary axis corresponding to the limb member 213 is parallel to theprimary axis. In some cases, one or more of the limb members can be anextension of the joint member 211. For example, if the medical device isa stapler with two jaws, the primary axis can correspond to the jointmember axis 198A of the stapler or where the two jaws meet and thesecondary axes can correspond to the axes of the limb members 198B,198C.

With continued reference to the embodiment illustrated in FIG. 2, asecond secondary axis can correspond to the axis 198C of the second limbmember 210 (or upper jaw) and is not parallel to the joint member axis198A or the first axis 198B in a first configuration. The second limbmember 210 can be moveable relative to the joint member 211 and thefirst limb member 213. In some cases, the second limb member 210 ishingeably connected to the joint member 211. In certain cases, the rangeof motion of the second limb member 210 can be an operating parameterinput into the system 100. In some embodiments, the range of motion ofthe second limb member 210 is limited by the hinge. In certainembodiments, a limb member (e.g., upper or lower jaw) can be affixed tothe joint member 211.

In a second configuration, such as when the multi-axis device isoperated or closed, the second limb member 210 can be parallel orapproximately parallel (e.g., within 5, 10, 15, or 20 degrees ofparallel (+/−a few degrees)) to the joint member 211. Thus, thesecondary axis 198C can change relative to the primary axis as thesecond jaw 210 moves. For example, if the multi-axis device 202 is astapler, the stapler 202 can be in a first configuration when thestapler in not activated. The stapler may be in a second (or stapling)configuration when the stapler in activated (e.g., the staples are beingapplied). In the stapling configuration, the second jaw 210 can beparallel (or approximately parallel) to the joint member 205. In suchcases, in the first configuration, the secondary axis 198C is notparallel to the primary axis 198A, and in the second configuration, thesecondary axis 198C is parallel (or approximately parallel) to theprimary axis 198A. Further, while moving from the first configuration tothe second configuration, the secondary axis 198C is not parallel to theprimary axis 198A.

In certain embodiments, both the first limb member 213 and the secondlimb member 210 are affixed to the joint member at different angles suchthat the primary axis and secondary axis or axes are all different(e.g., the joint member axis 198A, the first axis 198B, and the secondaxis 198C are all nonparallel). Thus, in some examples, there are atleast three nonparallel axes between the joint member 311 and limbmembers—the primary axis, the first secondary axis and the secondsecondary axis.

In some embodiments, the multi-axis medical device 202 can have morethan three axes and/or can have more than one primary axis. Forinstance, the joint member 211 can have more than one axis. As anexample, the joint member can include one or more bends or hinges (asdepicted in FIG. 10) wherefrom the joint member 211 can be articulatedrelative to a proximal shaft of the medical device.

Depicting Medical Device Placement, Trajectory, and Other Image GuidanceCues

In certain procedures, the system can provide image predictioninformation related to the medical devices as image guidance cues. Withcontinued reference to FIG. 2, the display 220 illustrates a 3D model ofa multi-axis medical device 202, a second medical device 223, a medicalimage 204, and image guidance cues. While FIG. 2 depicts the multi-axismedical device as a virtual stapler 202, the virtual medical device 202could include any two jaw device, including but not limited to atransector, resector, sealer, or grasper. Further, while the depictedmulti-axis device includes three axes, in some embodiments, themulti-axis device can include two, three, four, five or more axes. Forexample, the multi-axis device could include three limb membersextending out along nonparallel axes from a joint member.

In the illustrated embodiment, the medical device includes a jointmember 211 having a joint member axis 198A, a first limb member 213having a first axis 198B, and a second limb member 210 having a secondaxis 198C that is different from the joint member axis 198A and thefirst axis 198B. As mentioned above, the primary axis can correspond tothe joint member axis 198A or longitudinal axis of the joint member 211,and the secondary axes can correspond to the first axis 198B and thesecond axis 198C. In the illustrated embodiment of FIG. 2, one secondaryaxis (e.g., first axis 198B) is parallel to the primary axis and anothersecondary axis (e.g., second axis 198C) is not parallel to the primaryaxis. However, it should be understood that any of the secondary axiscan be parallel or not parallel to the primary axis. It should also benoted that, in some embodiments, a display object includes more than oneprimary axis.

In some embodiments, a trajectory (or trajectory indicator) of a displayobject can be determined and displayed. For instance, a trajectoryindicator can be displayed to indicate the trajectory of an associateddisplay object, if that display object is moved along a drive path. Insome embodiments, the drive path can be dynamic in that it can correlateto the path of the medical device when moved by a user. For instance,the system can determine that the display object (e.g., medical device)is being moved and, in response, can calculate a drive path whichcorresponding to the path the medical device would take if it continuedto move in the same direction. However, it should be understood that thedrive path can be predetermined or dynamic. For instance, the drive pathcan correspond to a path that is parallel to a primary axis of thedisplay object (akin to moving the medical device forward/backward). Asanother example, the drive path can correspond to a path that isperpendicular to a primary axis of the display object (akin to movingthe medical device up/down or side-to-side). Similarly, the drive pathcan be parallel or perpendicular to a secondary axis. In someembodiments, a display object can have a predetermined drive path whenthe system determines the display object is not moving and can have adynamic drive path when the system detects movement.

In a non-limiting example, the display object can be a multi-axismedical device (e.g., a virtual stapler 202) having a joint member 211,a first limb member 213, and second limb member 210. A trajectory can becalculated, determined, and displayed for the first limb member 213, thesecond limb member 210 and/or the joint member 211. For example, each ofthe trajectories can be calculated based on a drive path for the medicaldevice. In some cases, the trajectory for each member of the medicaldevice can correspond to the path that that portion of the medicaldevice (i.e., the joint member 211, the first limb member 213, and thesecond limb member) will follow if the medical device is moved along thedrive path. In some embodiments, the system 100 can display a trajectoryindicator for a distal portion of each limb member and/or joint memberof the medical device. However, it should be understood that atrajectory indicator can be displayed for any and/or all portions of adisplay object. Similarly, in some embodiments (as illustrated in FIG.2), a trajectory indicator may be displayed for some portions (e.g., thefirst limb member) and not displayed for other portions (e.g., thesecond limb member).

Referring to the non-limiting example above, trajectory indicators(e.g., 308A, 308B, 308C, and 308D of FIG. 3) can indicate a trajectoryalong an axis that is different from and parallel to the primary axis(non-limiting example: an axis corresponding to a distal portion of alimb member along the drive path). In addition, the trajectoryindicators can extend through or to a distal portion (e.g., the tip) ofthe medical device (e.g., a distal portion of the second limb member310). In some cases, the intersection indicators can correspond to, ortrack, an axis that intersects with a limb member axis (or secondaryaxis) at a distal portion of the limb member and is parallel to thejoint member axis (or primary axis) and/or the drive path.

In some embodiments, such as when both limb members are nonparallel tothe joint member, the trajectory indicator(s) can be determined based ona trajectory of the medical device along a primary axis (non-limitingexample: drive path that is parallel to the primary axis). In suchembodiments, the trajectory of the limb members can be determined basedon axes that are different from, but parallel to the primary axis andextend through a distal (or other) portion of the limb members.

While FIG. 2 illustrates a trajectory indicator 208 of a knife path ofthe first limb member 213, the system can additionally or alternativelyinclude a trajectory indicator of the second limb member 210 (as shownin FIG. 3), joint member 211, and/or a transecting knife (as shown inFIG. 3). In some embodiments, the trajectory indicator can indicate thetrajectory of the medical device along a primary axis and/or thesecondary axis. As mentioned above, the system can display one or moretrajectory indicators, which can indicate the trajectory of a displayobject if the display object is moved along a drive path. The trajectory(e.g., trajectory indicator 208) of the medical device 202 can bedepicted on the display 220 and can show the healthcare provider theprojected path that the medical device will take when moved.

A trajectory indicator can be displayed for any display object (forexample, virtual medical devices, affected regions, ghost displayobjects, image guidance cues, etc.) to indicate the trajectory of thedisplay object if moved along a drive path (e.g., corresponding to aprimary axis, a secondary axis, an axis parallel to the primary axis,etc.). In some embodiments, a trajectory indicator indicates thetrajectory of a display object if the display object is moved along anaxis parallel to the axis of another display object's axis or plane.

In some embodiments, as described below, a display object can begeometrically projected onto a plane-of-interest (i.e., the plane towhich that the display object is to be projected) to display a ghostdisplay object (e.g., an outline, cross section, or indication of thedisplay object on the plane-of-interest). In a non-limiting example, theghost display object can be a ghost medical device having a jointmember, a first limb member, and second limb member. The system candetermine and/or calculate a trajectory of the ghost medical device bydetermining the trajectory of the joint member, first limb member,and/or second limb member if the ghost medical device is moved along aprimary or secondary axis of the ghost medical device. It will beunderstood that the trajectory of multiple display objects can bedetermined and displayed simultaneously on screen 220, as described ingreater detail in the '274 Application. Further, it will be understoodthat the ghost display object can correspond to any display object. Forexample, the ghost display object can correspond a single axis medicaldevice, such as a needle, as described in greater detail in the '274Application.

In addition, in some examples, as described in greater detail below withrespect to FIG. 3, the guidance cues can include a ghost medical device230 (also referred to as a projected or mapped medical device) on theimage plane. In these examples, the system can additionally oralternatively display one or more trajectories (e.g., trajectoryindicators) of the ghost medical device, such as trajectory indicator231.

In some embodiments, to implement trajectory indicators, the imageguidance system can draw a number of rings about the joint member axis198A, the first axis 198B, the second axis 198C, or an axis parallel orperpendicular to the joint member axis 198A, the first axis 198B, or thesecond axis 198C and emanating from a distal (or other) end of a limbmember. The trajectory indicators can be extrapolated beyond the tip ofthe first limb member, second limb member or joint member. In exampleswhich include a ghost medical device, the image guidance system can drawsimilar trajectory indicators about the one or more axes correspondingto the ghost medical device 230. For example, ghost medical device 330can include a first and second limb member from which a trajectoryindicator can be drawn. Furthermore, the image guidance system can drawa number of rings about an axis of another medical device (for instance,the imaging device).

A healthcare provider can view and manipulate the emplacement of themedical device 242 and its trajectory (via an associated trajectoryindicator) before it enters the patient's body along a drive path. Insome embodiments, this is accomplished by the doctor positioning atrajectory indicator (e.g., virtual rings or cues) such that it iscoincident (or passes through) an ultrasound representation of a targetsuch as tissue that the doctor has spotted in the ultrasound.Positioning the trajectory indicator(s) such that it is properly aimedat the target can provide guidance to the doctor as she directs amedical device 242 into the body until it reaches its desired target ordestination. For example, if the doctor identifies tissue 212 in themedical image which needs attention, she can align the stapler 242 suchthat the trajectory indicators on display 220 intersect or otherwiseindicate that the medical device, if directed along the appropriateaxis, will reach the tissue 212.

In some embodiments, it may be advantageous for the doctor to positionthe trajectory indicator (e.g., virtual rings, cues, etc.) such that itis not coincident (or does not pass through) the ultrasoundrepresentation of a target. For example, with respect to a stapler, itmay be advantageous to align the stapler such that it is substantiallyparallel with and on or adjacent to the image plane, rather thanaligning the stapler such that it intersects the image plane. Byaligning the stapler in this manner, the healthcare provider can use themedical image 204 to see more of what tissue will be affected (e.g., theaffected region) when the stapler is activated.

The rings can, in some embodiments, be spaced at regular (e.g., 0.5, 1,or 2 cm) intervals to provide the healthcare provider with visual orguidance cues regarding the distance from the medical device tip to thetargeted anatomy. In some embodiments, the spacing of the rings canindicate other aspects of the data, such as the speed of the medicaldevice, the density of the tissue, the distance to a landmark, such asthe ultrasound data, or any other appropriate guidance data or property.In some embodiments, the rings or other trajectory indicators can extendbeyond the medical device tip, by a distance equal to the length of themedical device shaft. This way, the user knows if the medical device islong enough to reach the target, even before the tip(s) enter thepatient. That is, in some embodiments, if the rings do not reach thetarget with the tip still outside the body, then the tip will not reachthe target even when the entire device is inserted into the body. Insome embodiments, such as when it is preferable to aim the medicaldevice 202 at a target beyond the medical image 204, the trajectoryindicator 308 can extend to or even past the image plane.

Other display markers can be used to show trajectory, such as a dashed,dotted, or solid line, transparent medical device shaft, point cloud,wire frame, etc. In some embodiments, three-dimensional rings can beused and provide depth cues and obscure little of the medical image.Virtual rings or other virtual markers can be displayedsemi-transparently, so that they obscure less of the medical image thanan opaque marker would.

In addition, connecting lines 225 can also be drawn from aplane-of-interest (e.g., medical image plane) to a display object (e.g.,virtual medical device 202) or a display object plane (e.g., virtualmedical device plane). In some embodiments, the connecting lines extendorthogonally or perpendicularly from a plane-of-interest to a displayobject plane or a display object. In some embodiments, the connectinglines extend orthogonally or perpendicularly from a display object planeor a display object to a plane-of-interest. However, it will beunderstood that in some embodiments the connecting lines 225 are notorthogonal (or perpendicular) to any plane. In some embodiments, theconnecting lines 225 can be used to determine or display the differencein angle between a plane-of-interest and a display object plane. In somecases, the connecting lines can have an arc. The arc can be determinedby treating the intersection of the two display object trajectories asthe center of a sphere or circle and treating the connecting line as aperimeter of the circle or sphere.

Other prediction information can also be displayed as image guidancecues. For example, if a stapler is being tracked by the image guidancesystem, then a cutting plane corresponding to the trajectory of atransecting knife of the stapler can be displayed. Such a cutting plancan be coplanar with the transecting knife of the stapler and can extendfrom the transecting knife of the stapler (for instance, along the axisof the first member) to, for instance, a plane-of-interest. For example,the trajectory indicator of the transecting knife (e.g., cutting plane)can show where the stapler would cut if the doctor were to advance thestapler along a drive path. Similar prediction information can beestimable or determinable for graspers, vessel sealers, resectingdevices, transecting devices, scalpels cauterizers, lasers, and numerousother medical devices.

Furthermore, the data from two or more devices can be combined anddisplayed based on their relative emplacements or poses. For example,the system 100 can determine the emplacement of an image plane based onthe emplacement information of the ultrasound probe 222. Further, themedical image 204 can be displayed on the image plane with respect tothe virtual medical device 202 on the display 220 in a manner thatestimates the relative emplacements or poses of an ultrasound probe 222and the medical device 242. As illustrated in FIG. 2, the image guidancecues associated with the virtual medical device 202, includingtrajectory indicators, are shown spatially located with the medicalimage 204 on display 220.

Projecting a Ghost Display Object onto a Plane-of-Interest

In some embodiments, the system 100 can provide additional guidance cuesthat facilitate understanding of the relationship between the imageplane and the medical device(s). For example, display objects (e.g.,medical display objects (medical devices, medical images, affectedregions, etc.) or image guidance cues (trajectory indicators,intersection indicators, plane indicators, etc.)) can be geometricallyprojected onto a plane-of-interest such that a shadow, outline (e.g., adashed, dotted, or solid line), marking, transparent display object,point cloud, wire frame, or other indication of the display object isdisplayed on the plane-of-interest. In some embodiments, the projecteddisplay objects (e.g., projected medical devices, projected medicalimages, projected image guidance cues, etc.) are known collectively asghost display objects.

In some embodiments, the plane-of-interest is the plane to which thatthe display object is to be projected. For example, theplane-of-interest can be an image plane (e.g., the plane correspondingto a medical image), a medical device plane (e.g., the planecorresponding to a virtual medical device), or any other display objectplane or other desired plane. In some embodiments, the plane-of-interestcan include only a portion of a plane (e.g., the medical image portionof the image plane). However, it will be understood that aplane-of-interest can include more than a portion of any plane includingbut not limited to the image plane, medical device plane, affectedregion plane, or other display object plane. In some embodiments, theplane-of-interest can be determined dynamically based on the displayobject plane. Further, in some cases, the plane-of-interest can beselected to be a plane that is parallel to the display object plane. Forexample, the plane-of-interest can be selected as the display objectplane (as described below) that is parallel to the image plane of amedical image. However, it will be understood that in some cases, theplane-of-interest can be predetermined, such as a display object planethat bisects the display object into halves.

In some embodiments, the display object plane corresponds to any planethat is parallel to a primary or secondary axis of the display object.In some embodiments, the display object plane corresponds to a planethat bisects the display object (e.g., a virtual medical device) intoportions (e.g., left and right portions, top and bottom portions, etc.).In some cases, a display object plane can be determined dynamicallybased on the plane-of-interest. Further, in certain cases, the displayobject plane can be selected to be a plane that bisects the displayobject and that is parallel to the plane-of-interest. For example, thedisplay object plane can be selected as a plane (e.g., a planecorresponding to an axis of the display object) that is parallel to theplane-of-interest. Thus, the display object plane can change as thedisplay object moves throughout the system. However, it will beunderstood that in some cases, the display object plane can bepredetermined.

A ghost display object (e.g., projected display object) can be projectedonto a plane-of-interest using a variety of techniques. For example, theghost display object can be an orthographic projection of the displayobject onto the plane-of-interest. As a non-limiting example, thedisplay object is a virtual medical device and the plane-of-interest isan image plane. The system calculates a plurality of lines which areperpendicular to the image plane. For each perpendicular line thatintersects the virtual medical device (or a subset thereof), a point isdisplayed on the image plane at the location corresponding to theintersecting perpendicular line. As a result, a plurality of points aredisplayed on the image plane which correspond to the plurality ofperpendicular lines (or subset thereof) intersecting the virtual medicaldevice. The plurality of points can illustrate an indication of thevirtual medical device on the image plane. In some embodiments, theplurality of points can be referred to as a ghost, a shadow, an outline,a marking, a projection, or an indication of the virtual medical device.It should be understood that the number of perpendicular lines (and thenumber of corresponding points) can vary across embodiments. Forexample, in some embodiments, there can be almost infinite perpendicularlines such that a complete representation of the virtual medical deviceis displayed on the image plane. In some embodiments, the perpendicularlines can be spaced out at predefined intervals such that the ghostmedical device includes a dotted outline or dotted representation of themedical device. Similarly, in some cases, the subset of perpendicularlines used to generate the ghost display object can correspond to linesthat intersect with a particular portion of the virtual medical device,such as the edge thereby creating an outline on the plane-of-interest.

In some embodiments, a ghost display object can be projected onto aplane-of-interest using a virtual light source. Consider a virtual lightsource that is at least as large as the display object to be projectedor that covers an entire plane (e.g., a light source plane) positionedbehind the display object (e.g., distal the plane-of-interest relativeto the display object) and parallel to the plane-of-interest, and thatemits parallel light waves. In examples such as these, the ghost displayobject can be displayed on the plane-on-interest as a shadow of thedisplay object. Because the light source is infinitely wide (and can beinfinitely far away), the size of the ghost display object cancorrespond directly with the size of the display object. Further, itwill be understood that in some cases, the system can display an outlineof the shadow generated by the virtual light source or other indicator.

In some embodiments, a display object can be projected by identifyingpoints on a display object plane that correspond (e.g., intersect) tothe display and mapping (e.g., projecting, reproducing, copying) theidentified points onto a plane-of-interest. In some embodiments, theidentified points on the display object plane can include a plurality ofpoints (e.g., coordinates) that intersect with the corresponding displayobject. For instance, the plane of a virtual medical device can be aplane that bisects the virtual medical device. The plane can include aplurality of points that intersect with the virtual medical device. In anon-limiting example, to project the virtual medical device onto animage plane, at least some of the plurality of intersecting points canbe drawn on the image plane. In certain cases, as described above, theresulting ghost medical device can include an outline or cross-sectionof the virtual medical device.

In some embodiments, a display object can be projected by identifying ordetermining which points (e.g., coordinates) of the plane-of-interestare closest (e.g., nearest) to points (e.g., coordinates) of the displayobject or points of the display object intersecting the display objectplane (as described above). Each of the points corresponding to thedisplay object (or the display object plane) can be projected onto theplane-of-interest at the location of the points of the plane-of-interestthat were determined to be closest to the identified points of thedisplay object. In some embodiments, projecting can include determiningportions of the plane-of-interest that are closest to the displayobject. The display object can be projected onto those portions of theplane-of-interest that are closest to the display object. In someembodiments, projecting can include determining coordinates of across-sectional outline of the display object that are closest to theplane-of-interest.

As a non-limiting example, the ghost display object can be a ghostmedical device (for example, a ghost stapler) and the plane-of-interestcan be an image plane. The virtual medical device (for example, thevirtual stapler) can include a plurality of points on a stapler plane(e.g., a plane corresponding to an axis of the stapler, a plane thatbisects the stapler into approximate halves, a plane that is parallel tothe image plane, etc.). The system 100 determines which points on theimage plane are closest or nearest to the points of the virtual stapleron the stapler plane. The ghost medical device is projected by drawingthe points of the virtual medical device onto the determined nearestpoints of the image plane. In some embodiments, a ghost display objectis an outline or other indication of all or a portion of the displayobject. In some embodiments, the ghost display object is across-sectional outline of the display object or a portion thereof. Insome embodiments, a ghost display object is a projection of a full orpartial representation of the display object. For example, withcontinued reference to the non-limiting example, the ghost medicaldevice can include a representation of a first limb member, the secondlimb member, and/or joint member on the image plane.

As a non-limiting example, the ghost display object can be a ghostmedical image and the plane-of-interest can be a medical device plane.As mentioned above, the medical device plane can be a plane that bisectsthe left and right portions (or halves) or top and bottom portions (orhalves) of the virtual medical device or portion thereof, such as thejoint member. The virtual medical image can include a plurality ofpoints on the image plane. The system 100 determines which points on themedical device plane are closest or nearest to the points of the virtualmedical image. The ghost medical image is projected by drawing thepoints of the virtual medical image onto the determined nearest pointsof the medical device plane.

As a non-limiting example, the ghost display object is a ghost affectedregion and the plane-of-interest is an image plane. The affected regioncan include a plurality of points on a display object plane (forexample, a medical device plane, an affected region plane, etc.). Thesystem 100 can determine which points on the image plane are closest ornearest to the points of the affected region on the display objectplane. The ghost affected region can be projected by drawing the pointsfrom the affected region on the display object plane onto the determinednearest points of the image plane.

In some embodiments, a display object can be projected by determining anarc path and projecting the display object on the plane-of-interestalong the arc path. In some embodiments, the system can determine thearc path by first determining an intersection of a display objecttrajectory and the plane of interest. The system can use theintersection as the center of a sphere and determine an angle betweenthe display object trajectory and plane-of-interest. An arc path can bedetermined from the angle and each of the points corresponding to thedisplay object (as described above) can then be radially mapped to theplane-of-interest using the arc path. In some embodiments, a distancebetween each of the points corresponding to the display object and thepoint of intersection can be determined. In examples such as these, eachof the points can then be projected onto the plane-of-interest byplacing the points on the plane-of-interest at the same distances theywere on the display object plane.

As a non-limiting example, the ghost display object is a ghost medicaldevice and the plane-of-interest is an image plane. The system 100determines the intersection of the trajectory of the virtual medicaldevice and the image plane. As described above, the virtual medicaldevice can include a plurality of points on the medical device plane.The system 100 determines the distance between the intersection point(i.e., the center of the circle) and each of the plurality pointsdefining the medical device on the medical device plane. The system thenprojects the points of the medical device onto the image plane at thesame distance from the intersection point, but along the image plane.

FIG. 3 illustrates some embodiments of a medical device projected ontoan image plane. The ghost medical device 330 can include an outline orother indication of all or a portion of the virtual medical device 302.For example, the system can display a cross-sectional outline of thevirtual medical device 302 or a portion thereof. In the illustratedembodiment, the system displays a cross-sectional outline of a sideportion of the virtual medical device 302. However, it will beunderstood that the system can display more, less, or different parts ofthe medical device. For example, the system can display a projection ofa full or partial representation of the first limb member, the secondlimb member, the limb member, and/or the affected region onto the imageplane.

Although FIG. 3 depicts the ghost medical device 330 projected acrossthe entire image plane, in some examples, the ghost medical device 330is projected on only a portion of the image plane, such as within themedical image. However, it will be understood that the ghost medicaldevice 330 can be projected onto more than a portion of the image plane.Additionally, in some examples, only the affected region or a portion ofthe affected region is projected onto the plane-of-interest, such as animage plane or medical image, thereby allowing a healthcare provider tosee the region of tissue that would be affected if the medical device302 were positioned on the image plane. However, it will be understoodthat any portion of the medical device and/or affected region can beproject to the plane-of-interest.

In the illustrated embodiment, the system includes trajectory indicatorsof multiple display objects (e.g., virtual medical device 302 and ghostmedical device 330.) For example, FIG. 3 illustrates trajectoryindicator 308A and trajectory indicator 308B, which correspond to thefirst and second limb member, respectively, if the virtual medicaldevice is moved along a projected drive path (e.g., the primary axis.)Similarly, FIG. 3 illustrates trajectory 308C and trajectory 308Dcorresponding to the ghost first a second limb member, respectively, ifthe ghost medical device is moved along a projected drive path (e.g.,the primary axis of the ghost medical device). It should be noted that atrajectory can be displayed for any display object, including but notlimited to a virtual medical device, a ghost medical device, a medicalimage, a ghost medical image, an affected area, a ghost affected area,etc. In addition, the trajectory can be relative to any axis. Inembodiments where a display object has more than one axis, the displaycan display a trajectory indicator along a primary and/or secondaryaxis.

Trajectory indicators for ghost display objects can be calculated,determined, and displayed using a variety of techniques. In someembodiments, the trajectory indicators 308C, 308D can be projected ontothe plane-of-interest using techniques as described above with respectto projecting a display object onto a plane-of-interest. In someembodiments, the trajectory indicators 308C, 308D can be determined bycalculating a trajectory along an axis of the plane-of-interest. In someembodiments, the trajectory indicators 308C, 308D of the ghost medicaldevice 330 correspond to the trajectory indicators 308A, 308B of themedical device 302 such that the trajectory indicators 308C, 308Dintersect with trajectory indicators 308A, 308B. In some embodiments,the trajectory indicators 308C, 308D can indicator a trajectory along anaxis different than the trajectory axis of the trajectory indicators308A, 308B such that the trajectory indicators do not intersect. Forinstance, trajectory indicators 308C, 308D can extend along a secondaryaxis of the ghost medical device 330 and trajectory indicators 308A,308B can extend along a primary axis of the virtual medical device 302.

FIG. 4 illustrates some embodiments of a medical image (e.g., medicalimage 304) projected onto a medical device plane. The projected medicalimage 422 (also referred to as ghost medical image 422) can include anoutline or other indication of all or a portion of the medical image404. For example, the system can display a cross-sectional outline ofthe medical image 404 or a portion thereof. As a non-limiting, in someinstances, the ghost medical image 422 may be projected only within theaffected region of the virtual medical device 403. However, it will beunderstood that the ghost medical image 422 can be displayed in anylocation relative to the medical device plane. In some examples, theprojected medical image 422 can include a cube-shaped representation ora full or partial representation of the medical image. Further, theghost medical image and/or any other ghost display objects describedherein can be displayed in conjunction with any other embodimentsdescribed herein.

Intersection Indicators

Referring back to FIG. 2 and as mentioned above, the system can displayone or more trajectory indicators 231, 208. For instance, as describedabove, the system can display trajectory indicator 208 to indicate atrajectory of the medical device 202 along its projected drive path andthe system can display trajectory indicator 231 to indicate a trajectoryof the ghost display object 230 (e.g., ghost medical device) if movedalong its drive path. In some embodiments, because the ghost medicaldevice is a projected version of the virtual medical device, moving theghost medical device along its projected drive path means moving thevirtual medical device along its projected drive path. In some examples,as illustrated in FIG. 4, the system can also display trajectoryindicator 429 to indicate a trajectory another medical device 423 (e.g.,the ultrasound transducer) if directed along an associated drive path.In some examples, a trajectory of the transecting knife path can also bedisplayed. As mentioned above, the system can display a trajectoryindicator for any medical device or image and along any axis.

In addition to the trajectory indicators, the display 220 can includeanother image guidance cue in the form of one or more intersectionindicators, such as intersection indicator 209 (or intersectionindicators 309A, 309B of FIG. 3). The intersection indicators canindicate where one or more portions of the virtual medical device 202will intersect a plane (for example, the plane corresponding to themedical image 204) if the virtual medical device is moved along anassociated drive path and/or where one or more portions or axes of thevirtual medical device 202 intersect the plane. In some cases, at thepoint of intersection 209, the system can display an outline of aportion of the medical device and/or a box or other shape. For example,the system can display a cross-sectional outline of the medical device202 or a portion thereof. In the illustrated embodiment, the systemdisplays a cross-sectional outline 209 of the lower portion of themedical device 202 and a cross-sectional outline of a knife path 208.However, it will be understood that the system can display more, less,or different parts of the virtual medical device 202. For example, thesystem can display the upper portion at the point of intersection anddisplay a box around it as well, as desired. Further, additionalintersection indicators can be used to show the intersection of one ormore trajectories of the virtual medical device and the image plane ifthe virtual medical device is moved along a different drive path, suchas the secondary axis 198C. In addition, trajectory lines can bedisplayed extending away from the virtual medical device to show wherethe medical will go if moved forward along its principal or primary axisand/or one or more of the secondary axes. Furthermore, connecting lines231A, 231B can be drawn from the medical image 204 to the intersectionlocation.

In some examples, the intersection indicator 209 can indicate where thetrajectory of the virtual medical device, or trajectory indicator 208,intersects the trajectory of the ghost medical device 230 or trajectoryindicator 231. It should be noted that an intersection indicator can bedisplayed for the intersection of any two or more trajectories ortrajectory indicators, or for the intersection of any one or moreportions of the virtual medical device with the image plane. Forinstance, in some examples, as depicted in FIG. 4, the intersectionindicator 416 can indicate the intersection between the trajectory ofthe knife path, or trajectory indicator 408, and the trajectory of thesecond virtual medical device 423, or trajectory indicator 423.

When data related to two devices or medical devices are displayed withrelative emplacement, it can be difficult to orient their relativelocations if they do not intersect. In some embodiments, an imageguidance system will render relative location information. The relativelocation information may be shown with color (e.g., objects may berendered in brighter colors if they are closer), with renderingtechniques (e.g., objects may be rendered with transparency so that oneobject behind another may be visible, but visually appear behind thecloser object), with geometry (e.g., a geometric connector may be shownthat will allow the viewer to discern the relative relationships), orwith any other appropriate technique. For example, in some embodiments,if the intersection indicator is outside of the area of the ultrasoundslice, the image guidance system can draw geometry, such as a line (orrectangle) in the plane of the slice to indicate the needle's andmedical image's relative positions. In some embodiments, the relativelocations could also be represented using vertical and horizontalelements coplanar with the ultrasound or other medical image.

Depicting Affected Region and Other Information

Embodiments of the system can include image guidance cues as part of theimage guidance data to depict information related to the region orregions that will be affected by the use of medical devices. Forexample, in some embodiments, an image guidance cue displayed by theimage guidance system can include affected region information. Theillustrated embodiment of FIG. 5 shows a virtual multi-axis medicaldevice 502 and an additional 3D cue showing the affected region 526 forthe multi-axis medical device 502, indicating, for instance, a region oftissue that will be affected when the multi-axis device 502 isactivated. In some examples, the affected region can correspond to theinter-jaw area which can be a roughly triangular area with a curvededge, located between the multiple axes.

In some embodiments, the affected region can correspond to the volumelocated between the two jaws (i.e., the first limb member 513 and thesecond limb member 510). In some cases, the affected region can becalculated based at least in part on a distance from the hinge.Additional affected sub-regions can be calculated and displayed as well.For example, the plane at which the transecting knife cuts 514 (the lineshown in the middle portion of the bottom jaw 513 running generally leftto right) can be calculated and shown as well as the one or more planescorresponding to the one or more staples (in the illustrated embodiment,the five lines 527 or bands running transverse to the knife plane.

In some cases, the affected sub-regions can be transverse orperpendicular to each other. For example, an affected knife sub-regioncan be transverse to an affected staple sub-region. Further, any of thecues described herein can be displayed in conjunction with theaforementioned cues.

In some embodiments, such as when a multi-axis medical device is aimedat the medical image (e.g., a trajectory of the medical deviceintersects the medical image), the system can display the intersectionof the affected region or trajectory of the affected region along adrive path with the medical image (or image plane). Thus, in someembodiments, such as when the medical device is roughly parallel withthe image plane (or the medical device trajectory does not intersectwith the medical image) the system can project the affected region, orportions thereof, onto the image plane. In examples such as these, theghost affected region can be a roughly triangular area, which cancorrespond to the area or volume between the limb members that isprojected onto the image plane or medical image. In certain embodiments,such as when the medical device is roughly perpendicular (e.g., with 45degrees of being perpendicular) to the image plane (or the medicaldevice trajectory intersects with the medical image), the system candisplay an intersection indicator on the image plane that corresponds tothe intersection of the trajectory of the first limb member and themedical image or the intersection of the first limb member with themedical image. In addition, an intersection indicator can be displayedat the intersection of the trajectory of the second limb member and themedical image or the intersection of the second limb member with themedical image.

In some embodiments, the system can use the operating parameters of themedical device 502 and/or measured parameters to determine the affectedregion (and display the displayed affected region 526). For example, theaffected region's size (e.g., girth and length) can be dependent on thelength of the first limb member 513, the length of the second limbmember 510, the range of movement of the first limb member, the range ofmovement of the second limb member, etc. These dimensions and operatingranges can be manually entered and/or automatically computed by theguidance system based on device or operating parameters, such as, butnot limited to, the medical device make and model, power settings, andthe like. Similarly, the system can use measured parameters to determinethe affected region, such as, but not limited to, range of motion of thelimb member(s), distance between the limb members, available spacewithin the body of the patient, dimensions of tissue to be stapled, etc.For example, in the context of a stapler, the stapler may have a normalrange of motion (known as default settings) that can be adjusted basedat least in part on the size of the area to be stapled. In someembodiments, the measured parameters can be received in real-time asreal-time data. For instance, as distance between the stapler jaws isincreased or decreased, the system can display an adjusted affectedregion. In some examples, the system can use one or more formulas,look-up-tables, fixed or default values, or any other appropriateavailable information, etc. to determine the affected region.

In addition, the system can determine affected regions prior tooperating the medical device and/or during operation of the medicaldevice. For example, prior to operating the medical device, the systemcan determine one or more affected regions and/or during operation ofthe medical device, the system can determine one or more dynamicaffected regions. In some embodiments, the affected region is staticuntil operation of the medical device and then change as the device isoperated. In certain embodiments, the system may rely more on operatingparameters of the medical device to determine an initial affected region(i.e., the starting distance between the jaws of the medical device) andmeasured parameters to determine a dynamic affected region (i.e., as thejaws are opening or closing). However, it will be understood thatoperating parameters and/or measured parameters can be used to determinethe initial affected region and/or the dynamic affected region. In someinstance, the medical device 300 is inserted into and/or is moved aroundinside the patient in a closed or partially closed state in order toreduce the likelihood of causing damage with the medical device. In suchexamples, the display affected region can correspond to the smallerregion located between the first limb member and the second limb memberor the displayed affected area can be displayed as if the medical devicewas fully open.

In some circumstances, the operating parameters, measured parameters,formulas, a look-up-tables, fixed or default values, or otherinformation used to determine the affected regions may include someamount of error or variance. The variance may be due, for instance, touncertainty regarding the distance between the first limb member 313 andthe second limb member 310.

Returning to FIG. 2, the system can also determine the emplacement ofthe affected region. In some cases, the emplacement of the affectedregion can be based at least in part on the emplacement of some or allof the corresponding medical device (or virtual medical device), such asmedical device 242. For example, the system can receive emplacement datafrom one or more emplacement sensors associated with the medical device242 (non-limiting examples: coupled to or integrated with the medicaldevice 242, within an optical path of the medical device, etc.). Thesystem can use the emplacement data to determine the emplacement of thetracked medical device and/or the emplacement of the virtual medicaldevice 202 corresponding to the medical device 242. In some instances,the system can determine the emplacement of the medical device 242and/or virtual medical device 202 with respect to a point-of-viewlocation.

As yet another example, if the medical device is a stapler and theaffected region is a volume located between the first limb member andthe second limb member, the emplacement of the affected region can bebased at least in part on the emplacement of the stapler (or its virtualversion) or at least a portion of it, such as the location(s) on thestapler where the staples are located or portions of the first or secondlimb members. Specifically, in some embodiments, the affected region canbe centered at a location on the medical device, such as the portions onthe medical device that move when the medical device is operated. Insome examples, if multiple medical devices are used, the posed can bebased at least in part on the emplacement of the medical devices.

The system can display the affected region in a variety of ways.Furthermore, although the illustrated embodiment of FIG. 2 includes onlyone displayed affected region 206, it will be understood that one ormore affected regions can be displayed corresponding to each medicaldevice 242 that is displayed on the screen 220 and/or multiple affectedregions can be displayed corresponding to a single medical device 242.In some embodiments, the system can display a perspective view of theaffected region and/or non-perspective view, such as by displaying theaffected region on or with the medical image displayed in the 2D view.Some or all of the affected regions can be displayed as desired. In someembodiments, the portion of the affected region that is displayed can bereferred to as the displayed affected.

Although the illustrated embodiment of FIG. 2 refers to the affectedregion as a volume located between the first limb member and the secondlimb member, it will be understood that the affected region cancorrespond to a variety of medical procedures. For example, if acauterizer is tracked as part of an image guidance system, then theaffected region can correspond to a cauterization volume. If a laser istracked as part of the image guidance system, then the affected regioncan correspond to a projected laser path. If an ablation needed istracked as part of the image guidance system, then the affected regioncan correspond to an ablation volume. Similarly, the affected region cancorrespond to a biopsy volume, an electroporated volume, cryoablationvolume, laser ablation volume, high-frequency focused ultrasoundablation (HIFU) volume, external beam radiation therapy volume, anddrilling volume (where the display volume corresponds to the region ofbone and other tissue that the manually operated, or computer-controlleddrill would remove), depending on the type of medical instrument beingused.

As discussed above, in some embodiments, the system 100 projects theaffected region (or a portion of the affected region) of the virtualmedical device to the image plane or medical image. In some embodiments,the affected region (or a portion of the affected region) is projectedonto the plane-of-interest (for example, the image plane) in addition tothe ghost medical device. In some embodiments, the affected region (or aportion of the affected region) is projected onto the plane-of-interestwithout projecting a ghost medical device. In some examples, projectingthe ghost medical device and the affected region causes the image planeto be crowded. Thus, projecting only the affected region can make iteasier for the physician to see exactly what tissue will be affected.However, it will be understood that more than the affected region can beprojected onto the plane-of-interest.

Although FIG. 5 illustrates a multi-axis medical device having twodifferent axes, a multi-axis device can have any number of axes. Forexample, the multi-axis device can have three, four, five or more axes.As a non-limiting example, the multi-axis medical device can be athree-prong grasper (not shown). The three-prong grasper can have anaxis corresponding to each prong and an axis corresponding to its body.In this non-limiting example, the affected region corresponding to thethree-prong grasper can include the region between the three prongs, theregion corresponding to the volume that will be affected if the prongsare activated, etc.

Tracking a Medical Instrument

FIG. 6 illustrates an embodiment of a displayed medical device 602having visual tracking fiducials 632. As described above, in someembodiments, the position sensing unit 140 can include an opticaltracker and/or the one or more visually-detectable fiducials 632 can becoupled to the one or more medical devices. These markings 632 can betracked with the optical tracking system using, for instance,laparoscope video. Such marking 632 may be added to any instrumentsdescribed herein, including but not limited to a transducer, cauterizer,sealer, stapler, transector, resector, grasper, and any other medicaldevice.

It will be understood that the medical device 602 can be tracked in avariety of ways. As shown above, visual tracking fiducials 632 can beplaced on different portions of the medical device 602. In some cases,sensors, such as 5 DOF sensors or other sensors can be used. In someembodiments, multiple sensors can be coupled to the medical device 602.For example, one or more sensors can be located on the medical deviceproximal to the jaw hinge and/or proximal to a second hinge (e.g.,second hinge 1028 shown in FIG. 10), and/or one or more sensors can becoupled to a lower portion and/or the upper portion of the first and/orsecond limb member.

As described above, in some examples, a medical device 602 can also havemarkings such as bands around the shaft or jaws, for example, toindicate distance along the shaft or jaws. Healthcare providersperforming medical device procedures are often familiar with thesemarkings and can use them to help understand the spatial relationshipbetween the medical device and anatomy. In some embodiments, the makeand model of the medical device is known to the image guidance systemand the virtual medical device 602 displayed in display 220 can resemblemedical device 242. The features of medical devices that can be renderedin the scene include the overall shape (diameter, angles, crosssectional shape, curvature, etc.), color, distance markers, angle of thejaws, visuals or echogenic fiduciary markers, the state of deployableelements such as tines, paddles, anchors, resection loops, stiffening orsteerable sleeves, temperature, radiation, light or magnetic fieldsensors, lens, waveguides, fluid transfer channels, and the like.

In addition, the system can use a switch, sensor, or button or othermechanism to determine when the jaw has been closed. For example, aswitch or sensor can be near the handle or other actuator used to closethe jaw. Once activated, the system can update the virtual display toshow that the jaw has been closed. In some examples, as described above,jaw open or closure and distance between the jaws can be operatingparameters of the medical device that are used to display the virtualmedical device.

In some instances, a first medical device can be used to mark a path fora second medical instrument. Accordingly, the system 100 can track thefirst medical device such that the system can display the marked pathfor the second medical device. FIG. 7 illustrates an example of trackinga first medical device (e.g., a cauterizing device 702) which is used tocauterize (or mark) a path (e.g., a stapling, transecting, resecting,grasping path). In this example, the navigation software can assist thephysician to cauterize (or mark) tissue while inspecting the medicalimage relative to the tip of the cauterizing device. While using themedical image for guidance, the cauterizing device 702 and/or tip of thecauterizing device can be tracked and subsequently used as guidance fora second medical device (e.g., a stapler).

Any of the navigation features and/or guidance cues can also be used forwith the cauterizing device 730. For instance, the cauterizing device730 can have associated trajectory indicators, shadow bars, ghostcauterizing device, projections, tip-to-plane distance markers, and anyother guidance cues as described herein.

Medical Image can Traverse the Affected Region

In some embodiments, the medical image 804 can traverse the affectedregion 826 of the medical device 802. In such cases, the system 100 candetermine and display a point or area of intersection of the affectedregion 826 with the medical image 804.

In some examples, the system 100 can determine and display a point ofarea of intersection between a plane (for example, an image plane) orimage area and an affected region 826. As shown in FIG. 8, anintersection indicator 830 can be projected onto the image plane and/ormedical image 804 to illustrate the location on the medical image 804that the stapler will intersect if the stapler is activated (e.g.,closed). In some examples, the depicted intersection indicator canappear like a schematic of the multi-axis device superimposed onto theimage plane. In other examples, the intersection indicator 830 can bemarkings such as the guidance cues described above. Using theintersection indicator 830, the healthcare provider can identify where amedical device will intersect a plane (e.g., the image plane) if thestapler is operated.

Combining an Ultrasound Probe with a Two-Jaw Instrument

In some embodiments, an ultrasound transducer (for example, a miniatureultrasound transducer) can be integrated with a multi-axis instrumentsuch as a transecting stapler, grasper, or vessel sealer. A physicianmay use the integrated ultrasound transducer and jawed instrument toensure that the physician can view, in ultrasound, the internal tissuestructures before they are stapled, transected, resected, sealed, orgrasped, thereby helping the physician avoid critical structures (suchas blood vessels). As described above, a 3D viewing area can includeperspective views of each of a virtual integrated device, a medicalimage, navigation cues (e.g. affected region as shown in FIG. 5) orother cues described herein. In some examples, the integrated device caninclude one or more sensors located at different locations on thedevice. For example, a sensor can be embedded inside, or affixed to oneor more of the joint member, the first limb member, the second limbmember, and the ultrasound transducer (such as at the tip, along theshaft, or on a handle). Furthermore, the system can use the knownconfiguration of the device to determine the proper location between theaffected region and the medical image for display purposes.

FIG. 9 is a diagram illustrating an embodiment of a transducer mountedat the tip of the multi-axis instrument 922 such that the scan planecoincides with the inter-jaw plane. This is an elegant way to unify thetwo areas of interest mentioned previously so that both tissueinspection and tissue-affecting action (such as transection, stapling,resection, or grasping) take place in a single plane. As describedabove, a 3D viewing area can include perspective views of each at leasta portion of the medical image 904, at least a portion of the integrateddevice 922 (for example, the first limb member 913, the second limbmember 910, and/or the ultrasound transducer), a displayed affectedregion (not shown), and other image guidance cues (not shown).

Although FIG. 9 illustrates the transducer mounted to the tip of thefirst limb member, the transducer can be mounted anywhere on the medicaldevice. For example, the transducer can be mounted to the second limbmember 910 (such as at the tip, along the body, or near the hinge), thefirst limb member 913 (such as at the tip, along the body, or near thehinge), or the joint member (such as at the tip, along the body, ornear/on the hinge).

FIG. 10 illustrates an embodiment of an ultrasound transducer mounted toa multi-axis instrument. In the illustrated embodiment of FIG. 10, theintegrated device 1022 includes a transverse-mounted ultrasoundtransducer. The scanning plane of the transducer can be perpendicular tothe path of the transecting knife, the stapler axis (i.e., the jointmember axis), and/or the first member axis (i.e., the first axis). Thescanning plane of the transducer can also be perpendicular or parallelto the affected staple sub-regions (as described above).

FIG. 11 illustrates an embodiment of an ultrasound transducer mounted toa multi-axis instrument. In illustrated embodiment of FIG. 11, theintegrated device 1122 shows an ultrasound transducer mounted parallelto the rigid stapler surface and scanning “backwards.” This designattempts to provide better imaging of the area located exactly betweenthe jaws at the moment of stapling, transecting, resecting, sealing, orgrasping.

FIG. 12 illustrates an embodiment of an ultrasound transducer mounted toa multi-axis instrument. In illustrated embodiment of FIG. 12, a 2Dphased array miniature transducer is integrated with a multi-axisinstrument. The integrated device 1222 can scan a volumetric tissue areabetween the limb members 1210, 1213. A volumetric rendering method orconventional volume rendering could be used for real-time visualizationduring stapler guidance. It should be noted that the 2D phased arrayminiature transducer can be utilized in any of the previous examples.

It will be understood that the ultrasound transducer can be integratedwith the medical device in a variety of ways and that FIGS. 9-12 providenon-limiting examples. Further, the transducers can cover a variety ofangles. They can be attached to any portion of the multi-axis instrument(such as either the first limb member or the second limb member) andangled differently than shown. In addition, 2D phased arrays can be flatand thus integrate more seamlessly with tool designs than shown,avoiding local tissue displacement at the transducing element.

Flow Diagrams

FIG. 13 is a flow diagram illustrative of an embodiment of a routineimplemented by the system to determine and cause a display to displayone or more intersection indicators corresponding to an intersectionbetween a plane-of-interest and one or more trajectories of a medicaldevice. One skilled in the relevant art will appreciate that theelements outlined for routine 1300 can be implemented by one or morecomputing devices/components that are associated with the system 100,such as the position sensing unit 140, the image guidance unit 130,surgical system 149, an HMD, and/or the imager 150. Accordingly, routine1300 has been logically associated as being generally performed by thesystem 100. However, the following illustrative embodiment should not beconstrued as limiting. Furthermore, it will be understood that thevarious blocks described herein with reference to FIG. 13 can beimplemented in a variety of orders. For example, the system 100 canimplement some blocks concurrently or change the order as desired.Furthermore, it will be understood that fewer, more, or different blockscan be used as part of the routine 1300. For example, in someembodiments, the solid blocks are part of routine 1300 and the dottedblocks are not implemented. However, it will be understood that any ofthe blocks (and more or different blocks) can be implemented as part ofroutine 1300.

At block 1302, the system 100 receives first emplacement data associatedwith a first emplacement sensor and/or a first medical device. The firstemplacement data can be generated by the first emplacement sensor and/orby a position sensing unit. In some embodiments, the first emplacementsensor can be associated with the first medical device. For example, thefirst emplacement sensor can be associated with and/or attached to anultrasound transducer, as described above. In some embodiments, thefirst medical device is a multi-axis medical device. Furthermore, thefirst medical device has a joint member having a joint member axis, afirst limb member having a first axis, and a second limb member having asecond axis that is different from the joint member axis and the firstaxis. As described above, one or more emplacement sensors can beassociated with a first limb member, a second limb member, and/or ajoint member of the associated first medical device. The system canreceive emplacement data from each of these emplacement sensors.

At block 1304, the system 100 receives second emplacement dataassociated with a second emplacement sensor and/or a second medicaldevice. The second emplacement data can be generated by the secondemplacement sensor and/or by a position sensing unit. In someembodiments, the second emplacement sensor can be associated with thesecond medical device. For example, the second emplacement sensor can beassociated with and/or attached to a stapler, as described above withrespect to FIGS. 9-12. In some embodiments, the second medical device isan imaging device such as an ultrasound transducer.

At block 1306, the system 100 can determine an emplacement of a firstvirtual medical device based at least in part on the received firstemplacement data. As described above, the first virtual medical devicecan correspond to a multi-axis device such as a transecting stapler,grasper, or vessel sealer. In some embodiments, the virtual medicaldevice corresponds to a single-axis device such as an ablation probe.

In some embodiments, the system 100 can use the first emplacement dataand one or more characteristics of the associated first medical device(or a corresponding virtual medical device) to determine the emplacementof at least a portion of the first medical device. For example, the atleast a portion of the first medical device can include a portion thefirst limb member, a portion of the second limb member, or a portion ofthe joint member. As a non-limiting example, the characteristics of theassociated first medical device may indicate an emplacement of thesecond limb member relative to the associated first medical device.

Using the received emplacement data, the system 100 can determine theemplacement of the first medical device, for example, relative to knownlocation of emplacement sensor(s) on the first medical device. Forexample, the system 100 can use a known relationship between the firstemplacement data and the emplacement of the medical device (non-limitingexample: the second limb member extends 8 cm. away from the hinge at a45 degree angle relative to the first limb member) and/or use a knownrelationship between the emplacement of the emplacement sensor(s) and/orassociated medical device (or virtual medical device) and theemplacement of the medical device (non-limiting examples: the first limbmember begins 4 cm. from the tip of the first medical device (or firstvirtual medical device) and ends at the hinge of the first medicaldevice (or first virtual medical device), or the first limb memberextends 4 cm. in either direction from the ends of one of theemplacement sensors).

In addition, in certain embodiments, the system 100 can determine theemplacement of the first medical device in one or more coordinatesystems by mapping the first emplacement data, from one coordinatesystem to a second coordinate system. For example, the first emplacementdata may be received with respect to a first coordinate system, such asa position sensing coordinate system, and then mapped to a secondcoordinate system, such as a 3D scene coordinate system and/or a screencoordinate system. The emplacement of the first medical device can bedetermined with respect to one or more of the coordinate systems. Forexample, the emplacement of the first medical device can be determinedafter the first emplacement data has been mapped to the secondcoordinate system, such as the 3D scene coordinate system and/or thescreen coordinate system, or the emplacement of the first medical devicecan be determined for the first coordinate system, such as the positionsensing coordinate system, and then mapped to the 3D scene coordinatesystem and/or the screen coordinate system.

In certain embodiments, the system 100 can also use point-of-viewlocation (e.g., the location of the point-of-view for viewing the 3Dspace) to determine the emplacement of the first medical device forviewing. In some embodiments, as described in greater detail in '933Application, the point-of-view location can be a fixed location, such asa predetermined distance/angle from the screen 220 or stand 170 and or alocation configured by the user; or the point-of-view location can bydynamic. For example, the system can track a user in real-time anddetermine the point-of-view location based at least in part on thetracked location of the user.

In certain embodiments, the system 100 can also use an offset todetermine the emplacement of the first virtual medical device forviewing. For example, the system 100 can determine an initialemplacement of the first medical device in the 3D scene coordinatesystem and/or the screen coordinate system, and then apply an offset tothe initial emplacement and/or the system 100 can determine an initialemplacement of the first medical device in the position sensingcoordinate system and apply an offset to the initial emplacement priorto mapping the emplacement of the first medical device in the positionsensing coordinate system to the 3D scene coordinate system and/or thescreen coordinate system.

The offset can be made in one, or a combination of, coordinate systems,and/or with respect to one, or a combination of, axes. In certainembodiments, the offset can be made along a y-axis (up/down) of theposition sensing coordinate system, the 3D scene coordinate systemand/or the screen coordinate system. For example, the system 100 canadjust the y-coordinate (up/down coordinate) of the determinedemplacement of the first medical device (or the first emplacementsensor(s)) in the position sensing coordinate system by the offsetamount. When mapped to the 3D scene coordinate system and/or the screencoordinate system, the system 100 can use the adjusted emplacement. Asyet another example, the system 100 can adjust the y-coordinate (up/downcoordinate) of the determined emplacement of the first medical device(or the first emplacement sensor(s)) in the 3D scene coordinate systemand/or the screen coordinate system by the offset amount. Anycombination of the above-referenced examples can be used as desired.Furthermore, it will be understood that the offset can be made in anyone or any combinations of the coordinate systems and with reference toany one or any combination of the axes. For example, the adjustment canbe made along any one or any combination of the x-axis, y-axis, orz-axis.

Furthermore, the offset can be a predetermined offset and/or a dynamicoffset. In some embodiments, a predetermined offset can be used. Forexample, the system 100 can use a static offset based on an averageheight of males and/or females or average distance between elbows andhands, the height of the user, a distance between the user's elbow andeyes, expected location of a user with respect to the imaged volume,etc. In certain embodiments, the system 100 can use a dynamic offset,such as a determined emplacement of an HMD relative to one or moreemplacement sensors, position sensing region, and/or position sensingunit coordinate system. For example, the system 100 can determine theemplacement of an HMD relative to a medical device or imaged area andadjust the offset such that the medical image is always in view on thedisplays of an HMD. With continued reference to the example, if thewearer crouches down or turns to the side, the system 100 can determinethe change in relative emplacement between the HMD and the medicaldevice or imaged area and adjust the offset such that the medical imageremains in view in substantially the same emplacement.

In addition, it will be understood that the offset described herein withreference to the first medical device can be applied to any one or anycombination of the objects to be displayed and/or to all contents of thevirtual 3D scene. In some embodiments, the offset can be applied to someobjects to be displayed but not others.

At block 1308, the system 100 can determine an emplacement of a secondvirtual medical device based at least in part on the received secondemplacement data. As described above, the second virtual medical devicecan correspond to an imaging device such as an ultrasound transducer.The emplacement of the second virtual medical device can be determinedin a similar manner as described with respect to the first medicaldevice at block 1306.

At block 1310, the system 100 determines an emplacement of a medicalimage (e.g., medical image) based at least in part on received secondemplacement data. The medical image can be an intra-operative and/orreal-time medical image, such as a live ultrasound or intra-operative CTscan, or can be a pre-operative image, such as a pre-operative CT or MRIscan image. A real-time medical image (or real-time medical imagingstream) can refer to a medical image (or real-time medical imagingstream) received in real-time. The medical image received in real-timecan correspond to a live image, such as a live medical image generatedby an ultrasound or other image, such as a pre-operative orintra-operative CT image or MRI image that is communicated in real-time.

In some embodiments, the system 100 can use the second emplacement dataand one or more characteristics of the second emplacement sensor orassociated second medical device (or a corresponding second virtualmedical device) to determine the emplacement of the medical image. Forexample, the characteristics may indicate an emplacement of the medicalimage relative to the second emplacement sensor or associated secondmedical device (or second virtual medical imaging device).

Using this information, the system 100 can determine the emplacement ofthe medical image relative to the second emplacement sensor and/orassociated second medical device (or second virtual medical imagingdevice). For example, the system 100 can use a known relationshipbetween the second emplacement data and the emplacement of the medicalimage (non-limiting example: the medical image begins 2 cm. away fromthe of the second emplacement data location in a particular directionand ends 5 cm. away) and/or use a known relationship between theemplacement of the second emplacement sensor and/or associated secondmedical device (or second virtual medical imaging device) and theemplacement of the medical image (non-limiting examples: the medicalimage begins 4 cm. from the tip of the second medical device (or secondvirtual medical imaging device) and ends at the tip of the secondmedical device (or second virtual medical imaging device), or themedical image extends 2 cm. in either direction from the ends of thesecond emplacement sensor).

In addition, in certain embodiments, the system 100 can determine theemplacement of the medical image in one or more coordinate systems bymapping the first emplacement data, from one coordinate system to asecond coordinate system. For example, the second emplacement data maybe received with respect to a first coordinate system, such as aposition sensing coordinate system, and then mapped to a secondcoordinate system, such as a 3D scene coordinate system and/or a screencoordinate system. The emplacement of the medical image can bedetermined with respect to one or more of the coordinate systems. Forexample, the emplacement of the medical image can be determined afterthe second emplacement data has been mapped to the second coordinatesystem, such as the 3D scene coordinate system and/or the screencoordinate system, or the emplacement of the medical image can bedetermined for the first coordinate system, such as the position sensingcoordinate system, and then mapped to the 3D scene coordinate systemand/or the screen coordinate system.

In certain embodiments, the system 100 can also use point-of-viewlocation (e.g., the location of the point-of-view for viewing the 3Dspace) to determine the emplacement of the medical image for viewing. Insome embodiments, as described in greater detail in '933 Application,the point-of-view location can be a fixed location, such as apredetermined distance/angle from the screen 220 or stand 170 and or alocation configured by the user; or the point-of-view location can bydynamic. For example, the system can track a user in real-time anddetermine the point-of-view location based at least in part on thetracked location of the user.

As described above with respect to block 1306, in certain embodiments,the system 100 can also use an offset to determine the emplacement ofthe medical image for viewing. For example, the system 100 can determinean initial emplacement of the medical image in the 3D scene coordinatesystem and/or the screen coordinate system, and then apply an offset tothe initial emplacement and/or the system 100 can determine an initialemplacement of the medical image in the position sensing coordinatesystem and apply an offset to the initial emplacement prior to mappingthe emplacement of the medical image in the position sensing coordinatesystem to the 3D scene coordinate system and/or the screen coordinatesystem.

The offset can be made in one, or a combination of, coordinate systems,and/or with respect to one, or a combination of, axes. In certainembodiments, the offset can be made along a y-axis (up/down) of theposition sensing coordinate system, the 3D scene coordinate systemand/or the screen coordinate system. For example, the system 100 canadjust the y-coordinate (up/down coordinate) of the determinedemplacement of the medical image (or the first emplacement sensor) inthe position sensing coordinate system by the offset amount. When mappedto the 3D scene coordinate system and/or the screen coordinate system,the system 100 can use the adjusted emplacement. As yet another example,the system 100 can adjust the y-coordinate (up/down coordinate) of thedetermined emplacement of the medical image (or the first emplacementsensor) in the 3D scene coordinate system and/or the screen coordinatesystem by the offset amount. Any combination of the above-referencedexamples can be used as desired. Furthermore, it will be understood thatthe offset can be made in any one or any combinations of the coordinatesystems and with reference to any one or any combination of the axes.For example, the adjustment can be made along any one or any combinationof the x-axis, y-axis, or z-axis.

Furthermore, the offset can be a predetermined offset and/or a dynamicoffset. In some embodiments, a predetermined offset can be used. Forexample, the system 100 can use a static offset based on an averageheight of males and/or females or average distance between elbows andhands, the height of the user, a distance between the user's elbow andeyes, expected location of a user with respect to the imaged volume,etc. In certain embodiments, the system 100 can use a dynamic offset,such as a determined emplacement of an HMD relative to one or moreemplacement sensors, position sensing region, and/or position sensingunit coordinate system. For example, the system 100 can determine theemplacement of an HMD relative to a medical device or imaged area andadjust the offset such that the medical image is always in view on thedisplays of an HMD. With continued reference to the example, if thewearer crouches down or turns to the side, the system 100 can determinethe change in relative emplacement between the HMD and the medicaldevice or imaged area and adjust the offset such that the medical imageremains in view in substantially the same emplacement.

In addition, it will be understood that the offset described herein withreference to the medical image can be applied to any one or anycombination of the objects to be displayed and/or to all contents of thevirtual 3D scene. In some embodiments, the offset can be applied to someobjects to be displayed but not others.

At block 1312, the system 100 can determine a first intersection betweenan image plane and a trajectory of the first limb member along a drivepath. As described above with respect to FIGS. 2 and 3, a trajectory canbe calculated, determined, and displayed for the first limb member 213,the second limb member 210 and/or the joint member 211. For example,each of the trajectories can be calculated based on a drive path for themedical device. With respect to the first limb member, the trajectorycan correspond to the path that the first limb member will follow if themedical device is moved along the drive path.

In some examples, trajectory indicators can indicate a trajectory alongan axis that is different from and parallel to the primary axis(non-limiting example: an axis corresponding to a distal portion of thefirst limb member along the drive path). In addition, the trajectoryindicators can extend through a distal portion (e.g., the tip) of themedical device (e.g., a distal portion of the first limb member.

The first intersection indicator can indicate where the first limbmember of the first virtual medical device will intersect the imageplane (e.g., the plane corresponding to the medical image) if thevirtual medical device is moved along an associated drive path and/orwhere one or more portions or axes of the first virtual medical deviceintersect the image plane. As described above, the drive path can bepredetermined or dynamic.

At block 1314, the system 100 can determine a second intersectionbetween an image plane and a trajectory of the second limb member alongthe drive path. The second intersection indicator can be determinedusing the same techniques as described above and described with respectto block 1312. With respect to the second limb member, the trajectorycan correspond to the path that the second limb member will follow ifthe medical device is moved along the drive path. In some embodiments,the trajectory (e.g., the trajectory indicators) can extend through adistal portion (e.g., the tip) of the medical device (e.g., a distalportion of the second limb member.)

The second intersection indicator can indicate where the second limbmember of the first virtual medical device will intersect the imageplane (e.g., the plane corresponding to the medical image) if the firstvirtual medical device is moved along an associated drive path and/orwhere one or more portions or axes of the first virtual medical deviceintersect the image plane.

As described in greater detail above with reference to FIGS. 2 and 3,the system 100 can determine the points of intersection between one ormore trajectories and/or trajectory indicators, one or more axes (orportions) of one or more display objects (for example, a virtual medicaldevice, a ghost display object, an affected region, etc.) and/or one ormore planes (e.g., the virtual medical device plane, the image plane,etc.) based on the emplacement information of one or more medicaldevices.

In a non-limiting example, one or more intersection indicators can bedetermined and displayed for an intersection between a trajectory of amedical device (for example, a trajectory of the first limb member of astapler) and a plane-of-interest (for example, an image plane). One ormore intersection indicators can also be displayed for an intersectionbetween a trajectory of the second limb member along a drive path (forexample, an axis parallel to the primary axis and intersecting thesecond limb member) and a plane-of-interest (for example, an imageplane).

In a non-limiting example, one or more intersection indicators can bedisplayed for an intersection between a trajectory of a medical device(for example, a stapler, an ultrasound transducer, medical imagingdevice, etc.) and an affected region (or trajectory of an affectedregion)

In a non-limiting example, one or more intersection indicators can bedisplayed for an intersection between a trajectory of a medical device(for example, a stapler, an ultrasound transducer, medical imagingdevice, etc.) and an axis, trajectory, or plane of a transecting knife.

In a non-limiting example, one or more intersection indicators can bedisplayed for an intersection between a trajectory of a ghost displayobject (for example, a ghost medical device, a ghost affected region, aghost medical image, etc.), a stapler, an ultrasound transducer, medicalimaging device, etc., and an a trajectory of another display object (forexample, a medical device, an affected region, a medical image, etc.).

At block 1316, the system 100 can cause a display to display a view of avirtual 3D scene including a perspective rendering of at least a portionof the first virtual medical device based at least in part on thedetermined emplacement of the first virtual medical device. In someembodiments, the 3D rendering of the at least a portion of the firstvirtual medical device includes at least a portion of the first limbmember and at least a portion of the second limb member. However, itwill be understood that the 3D rendering of the at least a portion ofthe first virtual medical device can include any portions of the firstvirtual device.

As described above, the perspective view or rendering the medical imagecan be determined and displayed based at least in part on apoint-of-view location. The point-of-view location can be a fixedpoint-of-view location or a dynamic point-of-view location. For example,the point-of-view location can be with respect to a set location infront of the display and/or can be based on a tracked location of thedisplay, the HMD, or the user. In some embodiments, the point-of-viewlocation is determined based at least in part on the relativeemplacement of an HMD or user with respect to the position sensingregion or imaged volume, or the emplacement of an HMD within a positionsensing coordinate system. In embodiments, in which the point-of-viewlocation is based on a tracked location, the system 100 can enable theuser to view different perspective views of the virtual 3D scene basedon the changing emplacement of the tracked object (HMD, user) relativeto the medical devices 145, 155 or the position sensing region.

In certain embodiments, the system 100 can determine multipleemplacements for the first virtual medical device. For example, thesystem 100 can determine the emplacement for the medical image for aright-eye view and a left-eye view of a stereoscopic display, such for aHMD. In this way, each display for the HMD can display the medical imagefrom a slightly different perspective corresponding to a right-eye viewand a left-eye view, etc.

At block 1316, the system 100 can include the medical image in thevirtual 3D scene and cause the display to concurrently display aperspective rendering of at least a portion of the second virtualmedical device based at least in part on the determined emplacement ofthe second virtual medical device, similar to the display of the medicaldescribed above with reference to block 1314. In some embodiments, thedisplay of the second virtual medical device can be based at least inpart on dimensions of a corresponding real medical device (non-limitingexamples: size, shape, or other appearance). As described above withrespect to block 1314, the perspective view or rendering the medicalimage can be determined and displayed based at least in part on apoint-of-view location.

At block 1318, the system 100 can cause a display to display a view of avirtual 3D scene including a perspective rendering of at least a portionof the medical image based at least in part on the determinedemplacement of the medical image. As described above with respect toblock 1314, the perspective view or rendering the medical image can bedetermined and displayed based at least in part on a point-of-viewlocation.

At blocks 1322 and 1324, the system 100 can cause a display to displaythe first and second trajectory indicators. As described above, ahealthcare provider can use the displayed trajectory indicators to viewand manipulate the emplacement of the medical device and its trajectorybefore it enters the patient's body along a drive path. In someembodiments, this is accomplished by the doctor positioning a trajectoryindicator such that it is coincident (or passes through) an ultrasoundrepresentation of a target such as tissue that the doctor has spotted inthe ultrasound. Positioning the trajectory indicator(s) such that it isproperly aimed at the target can provide guidance to the doctor as shedirects a medical device into the body until it reaches its desiredtarget or destination. For example, if the doctor identifies tissue inthe medical image which needs attention, she can align the medical suchthat the trajectory indicators on display intersect or otherwiseindicate that the medical device, if directed along the appropriateaxis, will reach the tissue.

At blocks 1326 and 1328, the system 100 can cause a display to displaythe first and second intersection indicators. As described above, usingthe intersection indicators, the healthcare provider can identify wherea medical device (e.g., a multi-axis device) will intersect the plane(e.g., the image plane) if moved along a drive path (e.g., a primary, asecondary axis, axis parallel to the primary or secondary axis, etc.).Based on the desired placement of the medical device, the healthcareprovider can change the position and orientation of the medical deviceas desired. Thus, the intersection indicators as well as the trajectoryindicators can aid a healthcare provider during the placement of amedical device. Furthermore, the system can calculate distances betweenthe planes and the tips of the medical devices or other portions of themedical devices. The calculated distances can be displayed and/or usedto display the medical devices.

In some embodiments, the system 100 can determine and display a point ofintersection between a plane of interest (for example, an image plane, amedical device plane, etc.) and one or more medical device trajectories.For example, the system can determine and display a point ofintersection between an image plane and a joint member trajectory. Insome embodiments, the system 100 can determine and display a point ofintersection between a medical device plane and another medical devicetrajectory. In some embodiments, the system 100 can determine anddisplay a point of intersection between a ghost display object (ortrajectory of a ghost display object) and a plane (for example, an imageplane, a medical device plane, etc.). In some embodiments, a point ofintersection can be determined and displayed between a ghost displayobject (or its trajectory) and a medical device plane. For example, ifthe ghost display object is a ghost medical device, an intersectionindicator can be displayed at the intersection of a trajectory of theghost joint member and the medical device plane. Alternatively or inaddition, an intersection indicator can be displayed at the intersectionof the trajectory of the ghost second limb member along the drive pathand the medical image. In some examples, the point(s) of intersectionare located on the image plane, within the medical image. In otherexamples, the point(s) of intersection are located on the image plane,outside of the medical image.

It will be understood that the various blocks described herein can beimplemented in a variety of orders, and that the system 100 canimplement one or more of the blocks concurrently and/or change theorder, as desired. For example, the system 100 can concurrently receivethe emplacement data from different sources, concurrently receive themedical image, or receive the data in any order. Similarly, the system100 can concurrently determine the emplacement of the medical imageand/or one or more virtual medical devices, etc.

Furthermore, it will be understood that fewer, more, or different blockscan be used as part of the routine 1300. For example, the routine 1300can include blocks for receiving emplacement data associated withadditional emplacement sensors or medical devices, determiningemplacements of one or more medical devices, corresponding virtualmedical devices, other display objects, displays, and/or users. In someembodiments, the routine 1300 can include determining an emplacement ofa medical device associated with the medical image and determining theemplacement of the medical image based at least in part on thedetermined emplacement of the medical device (or corresponding virtualmedical device), and display the virtual medical device concurrentlywith the medical image. Furthermore, the system 100 can determine anddisplay a variety of image guidance cues, such as trajectory indicators,affected region indicators, intersection indicators, medical devices indifferent states or configurations, ghost display objects, as describedin greater detail above, and below with reference to FIG. 14. Inaddition, in some cases, the routine 1300 can omit certain blocks, suchas, but not limited to, blocks 1302, 1304, 1308, 1316, 1318, 1322,and/or 1324. For example, in some embodiments, the system may notdetermine emplacement of or display a portion of the second displayobject and/or may not display a portion of the first medical device,first trajectory indicator, and/or second trajectory indicator, etc.

FIG. 14 is a flow diagram illustrative of an embodiment of a routineimplemented by the system to project a display object onto aplane-of-interest to display a ghost display object. One skilled in therelevant art will appreciate that the elements outlined for routine 1400can be implemented by one or more computing devices/components that areassociated with the system 100, such as the position sensing unit 140,the image guidance unit 130, surgical system 149, an HMD, and/or theimager 150. Accordingly, routine 1400 has been logically associated asbeing generally performed by the system 100. However, the followingillustrative embodiment should not be construed as limiting.Furthermore, it will be understood that the various blocks describedherein with reference to FIG. 14 can be implemented in a variety oforders. For example, the system 100 can implement some blocksconcurrently or change the order, as desired. Furthermore, it will beunderstood that fewer, more, or different blocks can be used as part ofthe routine 1400. For example, in some embodiments, the solid blocks arepart of routine 1400 and the dotted blocks are not implemented. However,it will be understood that any of the blocks (and more or differentblocks) can be implemented as part of routine 1400.

At block 1402, as also described above with reference to block 1302 ofFIG. 13, the system 100 receives first emplacement data associated witha first emplacement sensor and/or a first medical device. In someembodiments, the first emplacement sensor can be associated with thefirst medical device. In some embodiments, the first medical device is amulti-axis medical device. Furthermore, the first medical device canhave a joint member having a joint member axis, a first limb memberhaving a first axis, and a second limb member having a second axis thatis different from the joint member axis and the first axis. As describedabove, one or more emplacement sensors can be associated with a firstlimb member, a second limb member, and/or a joint member of theassociated first medical device.

At block 1404, as also described above with reference to block 1304 ofFIG. 13, the system 100 receives second emplacement data associated witha second emplacement sensor and/or a second medical device.

At block 1406, the system 100 receives a medical image corresponding tothe second medical device.

At block 1408, as described similarly above with reference to block 1306of FIG. 13, the system 100 determines an emplacement of a first displayobject based at least in part on received first emplacement data. Insome embodiments, the first display object can be a multi-axis medicaldevice having a joint member, a first limb member, and a second limbmember. In some embodiments, the second display object can be anymedical display objects (e.g., virtual medical device, medical image,ghost display object, etc.) and/or any image guidance cue (e.g.,trajectory indicator, intersection indicator, plane indicator, ghostdisplay object, etc.).

At block 1410, the system 100 determines an emplacement of a seconddisplay object based at least in part on received second emplacementdata. In some embodiments, the second display object can be a medicalimage. In some embodiments, display object can be any medical displayobjects (e.g., virtual medical device, medical image, ghost displayobject, etc.) and/or any image guidance cue (e.g., trajectory indicator,intersection indicator, plane indicator, ghost display object, etc.).

At block 1412, as also described above with reference to block 1310 ofFIG. 13, the system 100 determines an emplacement of a medical image(e.g., medical image) based at least in part on received secondemplacement data.

At block 1414, the system 100 determines an emplacement of aplane-of-interest. In some embodiments, the plane-of-interest is theplane to which that the display object is to be projected. For example,the plane-of-interest can be an image plane (e.g., the planecorresponding to a medical image), a medical device plane (e.g., theplane corresponding to a virtual medical device), or any other displayobject plane or other desired plane. In some embodiments, theplane-of-interest can include only a portion of a plane (e.g., themedical image portion of the image plane). However, it will beunderstood that a plane-of-interest can include more than a portion ofany plane including but not limited to the image plane, medical deviceplane, affected region plane, or other display object plane. In someembodiments, the plane-of-interest can be determined dynamically basedon the display object plane. Further, in some cases, theplane-of-interest can be selected to be a plane that is parallel to thedisplay object plane. For example, the plane-of-interest can be selectedas the display object plane (as described below) that is parallel to theimage plane of a medical image. However, it will be understood that insome cases, the plane-of-interest can be predetermined, such as adisplay object plane that bisects the display object into halves.

In some embodiments, the display object plane corresponds to any planethat is parallel to a primary or secondary axis of the display object.In some embodiments, a virtual medical device plane can correspond to aplane that bisects the virtual medical device lengthwise or along itslongitudinal axis. For example, the plane can bisect the virtual medicaldevice into left and right portions (non-limiting examples: halves orother division), top and bottom portions (non-limiting examples: halvesor other division), upper-right and lower left portions, upper-left andlower right portions, etc. In some cases, a display object plane can bedetermined dynamically based on the plane-of-interest. Further, incertain cases, the display object plane can be selected to be a planethat bisects the display object and that is parallel to theplane-of-interest. For example, the display object plane can be selectedas a plane (e.g., a plane corresponding to an axis of the displayobject) that is parallel to the plane-of-interest. Thus, the displayobject plane can change as the display object moves throughout thesystem. However, it will be understood that in some cases, the displayobject plane can be predetermined. The emplacement of theplane-of-interest can be determined using a similar technique asdescribed above.

At block 1416, the system 100 determines an emplacement of a ghostdisplay object. The emplacement of the ghost display object can bedetermined using a variety of techniques, as described above. Forexamples, a ghost object can be projected onto the plane-of-interest. Aghost display object (e.g., projected display object) can be projectedonto a plane-of-interest using a variety of techniques. For example, theghost display object can be an orthographic projection of the displayobject onto the plane-of-interest. As a non-limiting example, thedisplay object is a virtual medical device and the plane-of-interest isan image plane. The system calculates a plurality of lines which areperpendicular to the image plane. For each perpendicular line thatintersects the virtual medical device (or a subset thereof), a point isdisplayed on the image plane at the location corresponding to theintersecting perpendicular line. As a result, a plurality of points aredisplayed on the image plane which correspond to the plurality ofperpendicular lines (or subset thereof) intersecting the virtual medicaldevice. The plurality of points can illustrate an indication of thevirtual medical device on the image plane. In some embodiments, theplurality of points can be referred to as a ghost, a shadow, an outline,a marking, a projection, or an indication of the virtual medical device.It should be understood that the number of perpendicular lines (and thenumber of corresponding points) can vary across embodiments. Forexample, in some embodiments, there can be almost infinite perpendicularlines such that a complete representation of the virtual medical deviceis displayed on the image plane. In some embodiments, the perpendicularlines can be spaced out at predefined intervals such that the ghostmedical device includes a dotted outline or dotted representation of themedical device. Similarly, in some cases, the subset of perpendicularlines used to generate the ghost display object can correspond to linesthat intersect with a particular portion of the virtual medical device,such as the edge thereby creating an outline on the plane-of-interest.

In some embodiments, a ghost display object can be projected onto aplane-of-interest using a virtual light source. Consider a virtual lightsource that is at least as large as the display object to be projectedor that covers an entire plane (e.g., a light source plane) positionedbehind the display object (e.g., distal the plane-of-interest relativeto the display object) and parallel to the plane-of-interest, and thatemits parallel light waves. In examples such as these, the ghost displayobject can be displayed on the plane-on-interest as a shadow of thedisplay object. Because the light source is infinitely wide (and can beinfinitely far away), the size of the ghost display object cancorrespond directly with the size of the display object. Further, itwill be understood that in some cases, the system can display an outlineof the shadow generated by the virtual light source or other indicator.

In some embodiments, a display object can be projected by identifyingpoints on a display object plane that correspond (e.g., intersect) tothe display and mapping (e.g., projecting, reproducing, copying) theidentified points onto a plane-of-interest. In some embodiments, theidentified points on the display object plane can include a plurality ofpoints (e.g., coordinates) that intersect with the corresponding displayobject. For instance, the plane of a virtual medical device can be aplane that bisects the virtual medical device. The plane can include aplurality of points that intersect with the virtual medical device. In anon-limiting example, to project the virtual medical device onto animage plane, at least some of the plurality of intersecting points canbe drawn on the image plane. In certain cases, as described above, theresulting ghost medical device can include an outline or cross-sectionof the virtual medical device.

In some embodiments, a display object can be projected by identifying ordetermining which points (e.g., coordinates) of the plane-of-interestare closest (e.g., nearest) to points (e.g., coordinates) of the displayobject or points of the display object intersecting the display objectplane (as described above). Each of the points corresponding to thedisplay object (or the display object plane) can be projected onto theplane-of-interest at the location of the points of the plane-of-interestthat were determined to be closest to the identified points of thedisplay object. In some embodiments, projecting can include determiningportions of the plane-of-interest that are closest to the displayobject. The display object can be projected onto those portions of theplane-of-interest that are closest to the display object. In someembodiments, projecting can include determining coordinates of across-sectional outline of the display object that are closest to theplane-of-interest.

Alternatively, or in addition, a medical image can be projected onto aplane of another medical device (e.g., a multi-axis medical device). Incertain embodiments, the system 100 projects at least a portion of themedical image onto a virtual medical device plane of a virtual medicaldevice based at least in part on the determined emplacement of thevirtual medical device and the determined emplacement of the medicalimage and/or medical device associated with the medical image (e.g., amedical imaging device such as an ultrasound transducer). The ghostmedical image can be displayed in a 2D view or 3D view and/or as aperspective view.

At steps 1418-1422, as similarly described above with respect to blocks1316-1320 of FIG. 13, the system 100 can display a 3D rendering of atleast a portion of the first display object, the second display object,or the medical image.

At step 1424, the system can display a 3D rendering of at least aportion of the ghost display object. In some embodiments, a ghostdisplay object is an outline or other indication of all or a portion ofthe display object. In some embodiments, the ghost display object is across-sectional outline of the display object or a portion thereof. Insome embodiments, a ghost display object is a projection of a full orpartial representation of the display object. For example, a ghostmedical device can include a representation of medical device having afirst limb member, the second limb member, and/or joint member on theimage plane. In some embodiments, the system can display a rendering ofat least a portion of the medical image projected onto theplane-of-interest.

It will be understood that the various blocks described herein can beimplemented in a variety of orders, and that the system 100 canimplement one or more of the blocks concurrently and/or change theorder, as desired. For example, the system 100 can concurrently receivethe emplacement data from different sources, concurrently receive themedical image, or receive the data in any order. Similarly, the system100 can concurrently determine the emplacement of the medical imageand/or one or more virtual medical devices, etc.

Furthermore, it will be understood that fewer, more, or different blockscan be used as part of the routine 1400. For example, the routine 1400can include blocks for receiving emplacement data associated withadditional emplacement sensors or medical devices, determiningemplacements of one or more medical devices, corresponding virtualmedical devices, other display objects, displays, and/or users. In someembodiments, the routine 1400 can include determining an emplacement ofa medical device associated with the medical image and determining theemplacement of the medical image based at least in part on thedetermined emplacement of the medical device (or corresponding virtualmedical device), and display the virtual medical device concurrentlywith the medical image. Furthermore, the system 100 can determine anddisplay a variety of image guidance cues, such as trajectory indicators,affected region indicators, as described above. In addition, in somecases, the routine 1400 can omit certain blocks, such as, but notlimited to, blocks 1402, 1404, 1406, 1410, 1412, 1420, and/or 1422. Forexample, in some embodiments, the system may not determine emplacementof or display a portion of the second display object and/or the medicalimage.

In addition, it will be understood that the various blocks describedherein with reference to FIG. 13 can be implemented in routine 1400, ina variety of orders. For example, the system 100 can implement some orall of various blocks of FIG. 13 in routine 1400 concurrently or changethe order as desired.

FIG. 15 is a flow diagram illustrative of an embodiment of a routineimplemented by the system to display an affected region of a medicaldevice. One skilled in the relevant art will appreciate that theelements outlined for routine 1500 can be implemented by one or morecomputing devices/components that are associated with the system 100,such as the position sensing unit 140, the image guidance unit 130,surgical system 149, and/or imaging unit 150. Accordingly, routine 1500has been logically associated as being generally performed by the system100. However, the following illustrative embodiment should not beconstrued as limiting. Furthermore, it will be understood that thevarious blocks described herein with reference to FIG. 15 can beimplemented in a variety of orders. For example, the system mayimplement some blocks concurrently or change the order, as desired.Furthermore, it will be understood that fewer, more, or different blockscan be used as part of the routine 1500. For example, in someembodiments, the solid blocks are part of routine 1500 and the dottedblocks are not implemented. However, it will be understood that any ofthe blocks (and more or different blocks) can be implemented as part ofroutine 1500.

At block 1502, as also described above with reference to block 1302 ofFIG. 13, the system 100 receives first emplacement data associated witha first emplacement sensor and/or a first medical device. In someembodiments, the first emplacement sensor can be associated with thefirst medical device. In some embodiments, the first medical device canbe a multi-axis medical device. Furthermore, the first medical devicecan have a joint member having a joint member axis, a first limb memberhaving a first axis, and a second limb member having a second axis thatis different from the joint member axis and the first axis. As describedabove, one or more emplacement sensors can be associated with a firstlimb member, a second limb member, and/or a joint member of theassociated first medical device.

At block 1504, the system 100 receives operating parameters of a medicaldevice. As described in greater detail above, the operating parameterscan include information regarding make and model, power, dimensions ofthe medical device (such as length or width of the first limb memberand/or the second limb member, angle data, distance from the hinge),range of movement data, etc. The operating parameters can be stored in anon-transitory, computer-readable medium associated with the system 100and/or can be stored in the medical device. The operating parameters canbe manually entered and/or automatically computed by the guidancesystem.

At block 1506, as also described above with reference to block 1306 ofFIG. 13, the system 100 can determine an emplacement of a first virtualmedical device based at least in part on the received first emplacementdata. As described above, the second virtual medical device cancorrespond to a multi-axis device such as a transecting stapler,grasper, or vessel sealer. In some embodiments, the virtual medicaldevice corresponds to a single-axis device such as an ablation probe.

At block 1508, the system 100 can determine an emplacement of anaffected region. In some embodiments, the system determines the affectedregion based at least in part on the operating parameters and/ormeasured parameters. As described previously, an affected region cancorrespond to a region of tissue that will be affected when the deviceis operated. As a non-limiting example, the first affected region cancorrespond to a volume located between the first limb member and thesecond limb member. In some examples, the first affected region can beone or more sub-regions of the region affected when the medical device(e.g., multi-axis medical device) is operated. For example, the affectedregion can correspond to the plane at which the transecting knife cutsand/or the one or more planes corresponding to the one or more staples.The emplacement of the affected region can be determined in a similarmanner as described above with respect to determining emplacement of amedical image, virtual medical device, ghost display object, etc.

At block 1510, similar to block 1316 of FIG. 13, the system 100 cancause the one or more displays to display a view of a virtual 3D sceneincluding a perspective rendering of at least a portion of the firstvirtual medical device based at least in part on the determinedemplacement of the first virtual medical device. In some embodiments,the 3D rendering of the at least a portion of the first virtual medicaldevice includes at least a portion of the first limb member and at leasta portion of the second limb member. However, it will be understood thatthe 3D rendering of the at least a portion of the first virtual medicaldevice can include any portions of the first virtual device.

At block 1512, the system 100 causes the one or more displays to displayat least a portion of the affected region. As described previously, thedisplayed affected region can be displayed in a 2D view or 3D viewand/or as a perspective view. In addition, as described in greaterdetail above, the displayed affected region can be displayed as avolume, area, and/or line. The displayed affected region can bewire-framed, transparent, semi-transparent, have varied opacity,brightness, and/or focus, include alternating bands/tiles, be textured,include solid or dashed lines, include spikes, etc. In certainembodiments, the at least a portion of the affected region correspondsto portions of the affected region that are unique to it, with respectto other affected regions. In some embodiments, the displayed affectedregion corresponds to at least a portion of the affected region that isco-located with at least a portion of a medical display object(non-limiting examples: a virtual medical device, medical image, etc.)or its trajectory, also referred to as the surface display region.

At block 1514, the system 100 causes the one or more displays to displayan intersection indicator indicating an intersection between an imageplane and a trajectory of the affected region along a drive path. Asdescribed above, for example with respect to block 1312 of FIG. 3, thedrive path can correspond to one of a plurality of paths. For example,in some embodiments, the drive path is parallel to a joint member axis.

In some embodiments (non-limiting examples illustrated in FIGS. 9-12),the medical image or medical image can traverse the affected region ofthe medical device. For example, the affected region can correspond to aregion of tissue that will be affected by a transecting knife and/orstaples when a multi-axis device is operated. In some embodiments, themedical image is traverse (or extends across) at least a portion of theaffected region.

With continued reference to FIG. 15, it will be understood that fewer ormore blocks can be included. For example, as described in greater detailabove, the system 100 can receive emplacement data from one or moresensors corresponding to one or more medical devices, determine aemplacement of the medical devices and/or corresponding virtual medicaldevices (as a non-limiting example, the emplacement can be determinedbased at least in part on a point-of-view location), cause one or moredisplays to display the virtual medical devices and/or perspective viewsthereof, determine emplacement of the affected regions with respect tothe virtual medical devices, determine emplacement of and display anmedical image, alter the display of the medical image, display themedical display objects in a 2D view, a 3D view, and/or a perspectiveview, etc. In some cases, the routine 1500 can omit certain blocks, suchas, but not limited to, blocks 1502, 1504, and/or 1510. For example, insome embodiments, the system may not display a portion of the virtualmedical device, etc.

In addition, it will be understood that the various blocks describedherein with reference to FIGS. 13 and/or 14 can be implemented inroutine 1500, in a variety of orders. For example, the system 100 canimplement some or all of various blocks of FIGS. 13 and/or 14 in routine1500 concurrently or change the order as desired.

FIG. 16 is a flow diagram illustrative of an embodiment of a routineimplemented by the system to determine emplacement of a virtual medicaldevice and a medical image based at least in part on receivedemplacement data associated with a medical device. One skilled in therelevant art will appreciate that the elements outlined for routine 1600can be implemented by one or more computing devices/components that areassociated with the system 100, such as the position sensing unit 140,the image guidance unit 130, surgical system 149, an HMD, and/or theimager 150. Accordingly, routine 1600 has been logically associated asbeing generally performed by the system 100. However, the followingillustrative embodiment should not be construed as limiting.Furthermore, it will be understood that the various blocks describedherein with reference to FIG. 16 can be implemented in a variety oforders. For example, the system 100 can implement some blocksconcurrently or change the order as desired. Furthermore, it will beunderstood that fewer, more, or different blocks can be used as part ofthe routine 1600. For example, in some embodiments, the solid blocks arepart of routine 1600 and the dotted blocks are not implemented. However,it will be understood that any of the blocks (and more or differentblocks) can be implemented as part of routine 1600.

In some embodiments, such as those illustrated in FIGS. 9-12, anultrasound transducer (for example, a miniature ultrasound transducer)can be integrated with a multi-axis instrument such as a transectingstapler, grasper, or vessel sealer. In examples such as these, anemplacement of a virtual medical device (for example, a stapler) and theemplacement of a virtual medical image can be determined based at leastin part on emplacement data associated with a single emplacement sensor.

At block 1602, as described above with reference to block 1303 of FIG.13, the system 100 receives first emplacement data associated with afirst emplacement sensor and/or a first medical device. The firstemplacement data can be generated by the first emplacement sensor and/orby a position sensing unit. In some embodiments, the first emplacementsensor can be associated with the first medical device (e.g., astapler).

At block 1604, as described above with reference to block 1306 of FIG.13, the system 100 can determine an emplacement of a first virtualmedical device based at least in part on the received first emplacementdata. As described above, the first virtual medical device cancorrespond to a multi-axis device such as a transecting stapler,grasper, or vessel sealer. In some embodiments, the virtual medicaldevice corresponds to a single-axis device such as an ablation probe.

At block 1606, similar to block 1308 of FIG. 13, the system 100 candetermine an emplacement of a second virtual medical device based atleast in part on the received first emplacement data. As describedabove, a single sensor and/or single emplacement data can be used todetermine the emplacement of the first medical device and the secondmedical device.

At block 1608, similar to block 1310 of FIG. 13, the system 100 candetermine an emplacement of a medical image based at least in part onthe received first emplacement data. As described above, a single sensorand/or single emplacement data can be used to determine the emplacementof the first medical device, the second medical device, and the medicalimage. As mentioned above, in some embodiments, because the medicaldevice (for example, the stapler) and the ultrasound transducer areintegrated together, both the emplacement of the virtual medical deviceand the emplacement of the virtual ultrasound transducer (and medicalimage) can be determined based at least at least in part on the receivedfirst emplacement data.

At steps 1610-1614, as similarly described above with respect to blocks1316-1320 of FIG. 13, the system 100 can display a 3D rendering of atleast a portion of the first display object, the second display object,or the medical image. However, it should be noted that each of thesedisplay objects are displayed based at least in part on the firstemplacement data.

It will be understood that the various blocks described herein can beimplemented in a variety of orders, and that the system 100 canimplement one or more of the blocks concurrently and/or change theorder, as desired. For example, the system 100 can concurrently receivethe emplacement data from different sources, concurrently receive themedical image, or receive the data in any order. Similarly, the system100 can concurrently determine the emplacement of the medical imageand/or one or more virtual medical devices, etc.

Furthermore, it will be understood that fewer, more, or different blockscan be used as part of the routine 1600. For example, the routine 1600can include blocks for receiving emplacement data associated withadditional emplacement sensors or medical devices, determiningemplacements of one or more medical devices, corresponding virtualmedical devices, other display objects, displays, and/or users. In someembodiments, the routine 1600 can include determining an emplacement ofa medical device associated with the medical image and determining theemplacement of the medical image based at least in part on thedetermined emplacement of the medical device (or corresponding virtualmedical device), and display the virtual medical device concurrentlywith the medical image. Furthermore, the system 100 can determine anddisplay a variety of image guidance cues, such as trajectory indicators,affected region indicators, as described above. In some cases, theroutine 1600 can omit certain blocks, such as, but not limited to,blocks 1602, 1606, 1612, and/or 1614. For example, in some embodiments,the system may not determine emplacement of or display a portion of thesecond display object.

In addition, it will be understood that the various blocks describedherein with reference to FIGS. 14 and 15 can be implemented in routine1600, in a variety of orders. For example, the system 100 can implementsome or all of various blocks of FIGS. 14 and 15 in routine 1600concurrently or change the order as desired.

Example Embodiments

Various example embodiments of methods, systems and non-transitorycomputer-readable medium relating to one or more intersection indicatorscorresponding to an intersection between a plane-of-interest and one ormore trajectories of a medical device can be found in the followingclauses:

Clause 1. A method for medical device navigation, the method comprising:receiving first emplacement data associated with a first medical device,the first medical device comprising a joint member having a joint memberaxis, a first limb member having a first axis, and a second limb memberhaving a second axis that is different from the joint member axis andthe first axis;

receiving second emplacement data associated with a second medicaldevice;

determining an emplacement of a first virtual medical devicecorresponding to the first medical device based at least in part on thereceived first emplacement data;

determining an emplacement of a second virtual medical devicecorresponding to the second medical device based at least in part on thereceived second emplacement data;

determining an emplacement of a medical image based at least in part onthe received second emplacement data;

determining a first intersection between an image plane and a trajectoryof the first limb member along a drive path that is parallel to thejoint member axis, wherein the image plane corresponds to the medicalimage;

determining a second intersection between the image plane and atrajectory of the second limb member along the drive path; and

causing one or more displays to concurrently display a perspective viewof:

-   -   a 3D rendering of at least a portion of the first virtual        medical device based at least in part on the determined        emplacement of the first virtual medical device, the 3D        rendering of the first virtual medical device including at least        the first limb member and the second limb member,    -   a 3D rendering of at least a portion of the second virtual        medical device based at least in part on the determined        emplacement of the second virtual medical device,    -   a 3D rendering of at least a portion of the medical image based        at least in part on the determined emplacement of the medical        image,    -   a first trajectory indicator corresponding to the trajectory of        the first limb member along the drive path,    -   a second trajectory indicator corresponding to the trajectory of        the second limb member along the drive path,    -   a first intersection indicator corresponding to the first        intersection, and    -   a second intersection indicator corresponding to the second        intersection.

Clause 2. A method for medical device navigation, the method comprising:

determining an emplacement of a first virtual medical devicecorresponding to a first medical device, the first medical devicecomprising a joint member having a joint member axis, a first limbmember having a first axis, and a second limb member having a secondaxis that is different from the joint member axis and the first axis;

determining an emplacement of a medical image;

determining a first intersection between an image plane corresponding tothe medical image and a trajectory of the first limb member along adrive path;

determining a second intersection between the image plane and atrajectory of the second limb member along the drive path; and

causing one or more displays to concurrently display a perspective viewof:

-   -   a 3D rendering of at least a portion of the medical image based        at least in part on the determined emplacement of the medical        image,    -   a first intersection indicator corresponding to the first        intersection, and    -   a second intersection indicator corresponding to the second        intersection.

Clause 3. The method of Clause 2, further comprising:

receiving first emplacement data associated with the first medicaldevice;

receiving second emplacement data associated with a second medicaldevice; and

determining an emplacement of a second virtual medical devicecorresponding to the second medical device based at least in part on thereceived second emplacement data,

wherein said determining the emplacement of the first virtual medicaldevice is based at least in part on the received first emplacement dataand said determining the emplacement of the medical image is based atleast in part on the received second emplacement data.

Clause 4. The method of any of the Clauses 2-3, wherein causing the oneor more displays to concurrently display a perspective view furthercomprises causing concurrent display of:

a 3D rendering of at least a portion of the first virtual medical devicebased at least in part on the determined emplacement of the firstvirtual medical device, the 3D rendering of the first virtual medicaldevice including at least the first limb member and the second limbmember, or

a 3D rendering of at least a portion of the second virtual medicaldevice based at least in part on the determined emplacement of thesecond virtual medical device.

Clause 5. The method of any of the Clauses 2-4, wherein causing the oneor more displays to concurrently display a perspective view furthercomprises causing concurrent display of:

a first trajectory indicator corresponding to the trajectory of thefirst limb member along the drive path, or

a second trajectory indicator corresponding to the trajectory of thesecond limb member along the drive path.

Clause 6. The method of any of the Clauses 2-5, wherein the drive pathis parallel to at least one of the joint member axis, the first axis, orthe second axis.

Clause 7. The method of any of the Clauses 2-6, wherein the drive pathis perpendicular to at least one of the joint member axis, the firstaxis, or the second axis.

Clause 8. The method of any of the Clauses 2-7, wherein the drive pathis predefined.

Clause 9. The method of any of the Clauses 2-8, wherein the drive pathis dynamic and is based at least in part on detected movement of firstmedical device or the second medical device.

Clause 10. The method of any of the Clauses 2-9, wherein a primary axisof the first medical device is the joint member axis.

Clause 11. The method of any of the Clauses 2-10, wherein a secondaryaxis of the first medical device is the first axis and/or the secondaxis.

Clause 12. The method of any of the Clauses 2-11, wherein the first axisis parallel to the joint member axis.

Clause 13. The method of any of the Clauses 2-12, wherein the first axisis different from the joint member axis.

Clause 14. The method of any of the Clauses 2-13, wherein the firstintersection indicator includes an outline of a cross-section of thefirst limb member.

Clause 15. The method of any of the Clauses 2-14, wherein the secondintersection indicator includes an outline of a cross-section of thesecond limb member.

Clause 16. The method of any of the Clauses 2-15, wherein the firstintersection indicator and/or the second intersection indicator includesa cross-sectional outline of a knife path and/or the first virtualmedical device, wherein the knife path corresponds to a transectingknife of the first medical device.

Clause 17. The method of any of the Clauses 2-16, wherein the firstintersection and/or the second intersection indicator includes anindication of at least a portion of the image plane.

Clause 18. The method of any of the Clauses 2-17, wherein the indicationof at least a portion of the image plane comprises a box representativeof a portion of the image plane.

Clause 19. The method of any of the Clauses 2-18, wherein the firsttrajectory indicator extends to at least a distal portion of the firstlimb member and corresponds to an axis along the drive path that isparallel to the joint member axis.

Clause 20. The method of any of the Clauses 2-19, wherein the firsttrajectory indicator corresponds to an axis that intersects with thefirst axis at a distal portion of the first limb member and is parallelto the joint member axis and the drive path.

Clause 21. The method of 20, wherein the second trajectory indicatorextends to at least a distal portion of the second limb member andcorresponds to an axis along the drive path that is different from andparallel to the joint member axis.

Clause 22. The method of any of the Clauses 2-213, wherein the secondtrajectory indicator corresponds to an axis that intersects with thesecond axis at a distal portion of the second limb member and isparallel to the joint member axis and the drive path.

Clause 23. The method of any of the Clauses 2-22, further comprisingdetermining a cutting path associated with the first medical device,wherein causing the one or more displays to concurrently display aperspective view further comprises causing concurrent display of thecutting path.

Clause 24. The method of Clause 23, wherein the cutting path correspondsto a trajectory of a knife of the first medical device.

Clause 25. The method of any of the Clauses 2-24, further comprising:

determining an emplacement of an affected region based at least in parton the determined emplacement of the first virtual medical device,

wherein causing the one or more displays to concurrently display aperspective view further comprises causing concurrent display of:

-   -   a 3D rendering of at least a portion of the affected region        based at least in part on the determined emplacement of the        virtual medical device, or    -   an affected region intersection indicator indicating an        intersection between a trajectory of the affected region along        the drive path.

Clause 26. The method of Clause 25, wherein the affected regioncorresponds to a volume located between the first limb member and thesecond limb member.

Clause 27. The method of Clause 25, wherein the at least a portion ofthe affected region comprises a stapling path of at least one staple ofthe first medical device.

Clause 28. The method of Clause 25, wherein the at least a portion ofthe affected region comprises a cutting path of a knife of the firstmedical device.

Clause 29. The method of Clause 25, further comprising determining anintersection between the affected region and the medical image.

Clause 30. The method of Clause 29, wherein the at least a portion ofthe affected region comprises the intersection between the affectedregion and the medical image.

Clause 31. The method of Clause 25, further comprising receivingoperating parameters corresponding to the first medical device, whereindetermining the emplacement of an affected region is based at least inpart on the received operating parameters.

Clause 32. The method of any of the Clauses 2-31, further comprising:

determining an emplacement of a plane-of-interest based at least in parton the determined emplacement of one of the medical image; and

determining an emplacement of a ghost medical device based at least inpart on the determined emplacement of the first virtual medical deviceand the determined emplacement of the plane-of-interest,

wherein causing the one or more displays to concurrently display aperspective view further comprises causing concurrent display of:

-   -   a 3D rendering of at least a portion of the ghost medical device        on the plane-of-interest based at least in part on the        determined emplacement of the ghost medical device.

Clause 33. The method of any of the Clauses 2-32, further comprising:

determining an emplacement of a plane-of-interest based at least in parton the determined emplacement of one of the first virtual medicaldevice; and

determining an emplacement of a ghost medical image based at least inpart on the determined emplacement of the medical image and thedetermined emplacement of the plane-of-interest,

wherein causing the one or more displays to concurrently display aperspective view further comprises causing concurrent display of:

-   -   a 3D rendering of at least a portion of the ghost medical image        on the plane-of-interest based at least in part on the        determined emplacement of the ghost medical image.

Clause 34. The method of any of Clauses 32 or 33, further comprisingprojecting the display object on the plane-of-interest to display theghost display object.

Clause 35. The method of Clause 34, wherein projecting comprisesdetermining at least one of portions, points, or coordinates of theplane-of-interest that are closest to the display object.

Clause 36. The method of Clause 34, wherein projecting comprisesdetermining an arc path corresponding to the display object and theplane-of-interest.

Clause 37. The method of Clause 34, wherein projecting comprisesdetermining which of a plurality of connecting lines intersect thedisplay object, wherein the plurality of connecting lines extendorthogonally from the plane-of-interest to the display object.

Clause 38. The method of Clause 34, wherein projecting comprises using avirtual light source that is at least as large as the display object tobe projected, wherein the light source is distal the plane-of-interestrelative to the display object.

Clause 39. The method of Clause 32, wherein the ghost display objectincludes a cross-section and/or outline of display object.

Clause 40. The method of Clause 32, wherein the ghost display objectincludes a cross-section and/or outline of display object.

Clause 41. A system comprising:

a computer system in communication with a display, the computer systemcomprising a computer processor and a non-transitory storage medium,wherein the computer system is configured to:

-   -   receive first emplacement data associated with a first medical        device, the first medical device comprising a joint member        having a joint member axis, a first limb member having a first        axis, and a second limb member having a second axis that is        different from the joint member axis and the first axis,    -   receive second emplacement data associated with a second medical        device;    -   determine an emplacement of a first virtual medical device        corresponding to the first medical device based at least in part        on the received first emplacement data,    -   determine an emplacement of a second virtual medical device        corresponding to the second medical device based at least in        part on the received second emplacement data,    -   determine an emplacement of a medical image based at least in        part on the received second emplacement data,    -   determine a first intersection between an image plane and a        trajectory of the first limb member along a drive path that is        parallel to the joint member axis, wherein the image plane        corresponds to the medical image,    -   determine a second intersection between the image plane and a        trajectory of the second limb member along the drive path, and    -   cause the display to concurrently display a perspective view of:        -   a 3D rendering of at least a portion of the first virtual            medical device based at least in part on the determined            emplacement of the first virtual medical device, the 3D            rendering of the first virtual medical device including at            least the first limb member and the second limb member,        -   a 3D rendering of at least a portion of the second virtual            medical device based at least in part on the determined            emplacement of the second virtual medical device,        -   a 3D rendering of at least a portion of the medical image            based at least in part on the determined emplacement of the            medical image,        -   a first trajectory indicator corresponding to the trajectory            of the first limb member along the drive path,        -   a second trajectory indicator corresponding to the            trajectory of the second limb member along the drive path,        -   a first intersection indicator corresponding to the first            intersection, and        -   a second intersection indicator corresponding to the second            intersection; and the display.

Clause 42. A system comprising:

a computer system in communication with a display, the computer systemcomprising a computer processor and a non-transitory storage medium,wherein the computer system is configured to:

-   -   determine an emplacement of a first virtual medical device        corresponding to a first medical device, the first medical        device comprising a joint member having a joint member axis, a        first limb member having a first axis, and a second limb member        having a second axis that is different from the joint member        axis and the first axis,    -   determine an emplacement of a medical image,    -   determine a first intersection between an image plane        corresponding to the medical image and a trajectory of the first        limb member along a drive path,    -   determine a second intersection between the image plane and a        trajectory of the second limb member along the drive path, and    -   cause the display to concurrently display a perspective view of:        -   a 3D rendering of at least a portion of the medical image            based at least in part on the determined emplacement of the            medical image,        -   a first intersection indicator corresponding to the first            intersection, and        -   a second intersection indicator corresponding to the second            intersection; and the display.

Clause 43. The system of the Clause 42, wherein the computer system isfurther configured to:

receive first emplacement data associated with the first medical device;

receive second emplacement data associated with a second medical device;

determine an emplacement of a second virtual medical devicecorresponding to the second medical device based at least in part on thereceived second emplacement data;

determine the emplacement of the first virtual medical device based atleast in part on the received first emplacement data; and

determine the emplacement of the medical image is based at least in parton the received second emplacement data.

Clause 44. The system of any of clauses 42 or 43, wherein the computersystem is further configured to cause the display to concurrentlydisplay a perspective view of:

a 3D rendering of at least a portion of the first virtual medical devicebased at least in part on the determined emplacement of the firstvirtual medical device, the 3D rendering of the first virtual medicaldevice including at least the first limb member and the second limbmember, or

a 3D rendering of at least a portion of the second virtual medicaldevice based at least in part on the determined emplacement of thesecond virtual medical device.

Clause 45. The system of any of clauses 42-44, wherein the computersystem is further configured to cause the display to concurrentlydisplay a perspective view of:

a first trajectory indicator corresponding to the trajectory of thefirst limb member along the drive path, or

a second trajectory indicator corresponding to the trajectory of thesecond limb member along the drive path.

Clause 46. The system of any of clauses 42-45, wherein the drive path isparallel to at least one of the joint member axis, the first axis, orthe second axis.

Clause 47. The system of any of clauses 42-46, wherein the drive path isperpendicular to at least one of the joint member axis, the first axis,or the second axis.

Clause 48. The system of any of clauses 42-47, wherein the drive path ispredefined.

Clause 49. The system of any of clauses 42-48, wherein the drive path isdynamic and is based at least in part on detected movement of firstmedical device or the second medical device.

Clause 50. The system of any of clauses 42-49, wherein a primary axis ofthe first medical device is the joint member axis.

Clause 51. The system of any of clauses 42-50, wherein a secondary axisof the first medical device is the first axis and/or the second axis.

Clause 52. The system of any of clauses 42-51, wherein the first axis isparallel to the joint member axis.

Clause 53. The system of any of clauses 42-52, wherein the first axis isdifferent from the joint member axis.

Clause 54. The system of any of clauses 42-53, wherein the firstintersection indicator includes an outline of a cross-section of thefirst limb member.

Clause 55. The system of any of clauses 42-54, wherein the secondintersection indicator includes an outline of a cross-section of thesecond limb member.

Clause 56. The system of any of clauses 42-55, wherein the firstintersection indicator and/or the second intersection indicator includesa cross-sectional outline of a knife path and/or the first virtualmedical device, wherein the knife path corresponds to a transectingknife of the first medical device.

Clause 57. The system of any of clauses 42-56, wherein the firstintersection and/or the second intersection indicator includes anindication of at least a portion of the image plane.

Clause 58. The system of clause 57, wherein the indication of at least aportion of the image plane comprises a box representative of a portionof the image plane.

Clause 59. The system of clause 45, wherein the first trajectoryindicator extends to at least a distal portion of the first limb memberand corresponds to an axis along the drive path that is parallel to thejoint member axis.

Clause 60. The system of clause 45, wherein the first trajectoryindicator corresponds to an axis that intersects with the first axis ata distal portion of the first limb member and is parallel to the jointmember axis and the drive path.

Clause 61. The system of clause 45, wherein the second trajectoryindicator extends to at least a distal portion of the second limb memberand corresponds to an axis along the drive path that is different fromand parallel to the joint member axis.

Clause 62. The system of clause 45, wherein the second trajectoryindicator corresponds to an axis that intersects with the second axis ata distal portion of the second limb member and is parallel to the jointmember axis and the drive path.

Clause 63. The system of any of clauses 42-62, wherein the computersystem is further configured to determine a cutting path associated withthe first medical device and cause the display to concurrently display aperspective view of the cutting path.

Clause 64. The system of clause 63, wherein the cutting path correspondsto a trajectory of a knife of the first medical device.

Clause 65. The system of any of clauses 42-64, wherein the computersystem is further configured to:

determine an emplacement of an affected region based at least in part onthe determined emplacement of the first virtual medical device; and

causing the display to concurrently display a perspective view of:

-   -   a 3D rendering of at least a portion of the affected region        based at least in part on the determined emplacement of the        virtual medical device, or    -   an affected region intersection indicator indicating an        intersection between a trajectory of the affected region along        the drive path.

Clause 66. The system of clause 65, wherein the affected regioncorresponds to a volume located between the first limb member and thesecond limb member.

Clause 67. The system of clause 65, wherein the at least a portion ofthe affected region comprises a stapling path of at least one staple ofthe first medical device.

Clause 68. The system of clause 65, wherein the at least a portion ofthe affected region comprises a cutting path of a knife of the firstmedical device.

Clause 69. The system of clause 65, wherein the at least a portion ofthe affected region comprises the intersection between the affectedregion and the medical image.

Clause 70. The system of any of clauses 42-69, wherein the computersystem is further configured to determine an intersection between theaffected region and the medical image.

Clause 71. The system of any of clauses 42-70, wherein the computersystem is further configured to receive operating parameterscorresponding to the first medical device and determine the emplacementof an affected region based at least in part on the received operatingparameters.

Clause 72. The system of any of clauses 42-70, wherein the computersystem is further configured to:

determine an emplacement of a plane-of-interest based at least in parton the determined emplacement of one of the medical image;

determine an emplacement of a ghost medical device based at least inpart on the determined emplacement of the first virtual medical deviceand the determined emplacement of the plane-of-interest; and

cause the display to concurrently display a perspective view of a 3Drendering of at least a portion of the ghost medical device on theplane-of-interest based at least in part on the determined emplacementof the ghost medical device.

Clause 73. The system of any of clauses 42-70, wherein the computersystem is further configured to:

determine an emplacement of a plane-of-interest based at least in parton the determined emplacement of one of the first virtual medicaldevice;

determine an emplacement of a ghost medical image based at least in parton the determined emplacement of the medical image and the determinedemplacement of the plane-of-interest; and

cause the display to currently display a perspective view of a 3Drendering of at least a portion of the ghost medical image on theplane-of-interest based at least in part on the determined emplacementof the ghost medical image.

Clause 74. The system of any one of clause 72 or 73, wherein thecomputer system is further configured to project the display object onthe plane-of-interest to display the ghost display object.

Clause 75. The system of the Clause 74, wherein the computer system isfurther configured to determine at least one of portions, points, orcoordinates of the plane-of-interest that are closest to the displayobject.

Clause 76. The system of any of clauses 74 or 75, wherein the computersystem is further configured to determine an arc path corresponding tothe display object and the plane-of-interest.

Clause 77. The system of any of clauses 74-76, wherein the computersystem is further configured to determine which of a plurality ofconnecting lines intersect the display object, wherein the plurality ofconnecting lines extend orthogonally from the plane-of-interest to thedisplay object.

Clause 78. The system of any of clauses 74-77, wherein the computersystem further comprises a virtual light source that is at least aslarge as the display object to be projected, wherein the light source isdistal the plane-of-interest relative to the display object.

Clause 79. The system of any of clauses 74-78, wherein the ghost displayobject includes a cross-section and/or outline of display object.

Clause 80. The system of any of clauses 74-79, wherein the ghost displayobject includes a cross-section and/or outline of display object.

Clause 81. A non-transitory computer-readable medium comprising computerexecutable instructions that when executed by one or more processorscause the one or more processors to:

-   -   receive first emplacement data associated with a first medical        device, the first medical device comprising a joint member        having a joint member axis, a first limb member having a first        axis, and a second limb member having a second axis that is        different from the joint member axis and the first axis,    -   receive second emplacement data associated with a second medical        device;    -   determine an emplacement of a first virtual medical device        corresponding to the first medical device based at least in part        on the received first emplacement data,    -   determine an emplacement of a second virtual medical device        corresponding to the second medical device based at least in        part on the received second emplacement data,    -   determine an emplacement of a medical image based at least in        part on the received second emplacement data,    -   determine a first intersection between an image plane and a        trajectory of the first limb member along a drive path that is        parallel to the joint member axis, wherein the image plane        corresponds to the medical image,    -   determine a second intersection between the image plane and a        trajectory of the second limb member along the drive path, and    -   cause the display to concurrently display a perspective view of:        -   a 3D rendering of at least a portion of the first virtual            medical device based at least in part on the determined            emplacement of the first virtual medical device, the        -   3D rendering of the first virtual medical device including            at least the first limb member and the second limb member,        -   a 3D rendering of at least a portion of the second virtual            medical device based at least in part on the determined            emplacement of the second virtual medical device,        -   a 3D rendering of at least a portion of the medical image            based at least in part on the determined emplacement of the            medical image,        -   a first trajectory indicator corresponding to the trajectory            of the first limb member along the drive path,        -   a second trajectory indicator corresponding to the            trajectory of the second limb member along the drive path,        -   a first intersection indicator corresponding to the first            intersection, and        -   a second intersection indicator corresponding to the second            intersection.

Clause 82. A non-transitory computer-readable medium comprising computerexecutable instructions that when executed by one or more processorscause the one or more processors to:

-   -   determine an emplacement of a first virtual medical device        corresponding to a first medical device, the first medical        device comprising a joint member having a joint member axis, a        first limb member having a first axis, and a second limb member        having a second axis that is different from the joint member        axis and the first axis,    -   determine an emplacement of a medical image,    -   determine a first intersection between an image plane        corresponding to the medical image and a trajectory of the first        limb member along a drive path,    -   determine a second intersection between the image plane and a        trajectory of the second limb member along the drive path, and    -   cause the display to concurrently display a perspective view of:        -   a 3D rendering of at least a portion of the medical image            based at least in part on the determined emplacement of the            medical image,        -   a first intersection indicator corresponding to the first            intersection, and        -   a second intersection indicator corresponding to the second            intersection.

Clause 83. The non-transitory computer-readable medium of clause 82,wherein the computer executable instructions further cause the one ormore processors to:

receive first emplacement data associated with the first medical device;

receive second emplacement data associated with a second medical device;

determine an emplacement of a second virtual medical devicecorresponding to the second medical device based at least in part on thereceived second emplacement data;

determine the emplacement of the first virtual medical device based atleast in part on the received first emplacement data; and

determine the emplacement of the medical image is based at least in parton the received second emplacement data.

Clause 84. The non-transitory computer-readable medium of clauses 82 or83, wherein the computer executable instructions further cause the oneor more processors to cause the display to concurrently display aperspective view of:

a 3D rendering of at least a portion of the first virtual medical devicebased at least in part on the determined emplacement of the firstvirtual medical device, the 3D rendering of the first virtual medicaldevice including at least the first limb member and the second limbmember, or

a 3D rendering of at least a portion of the second virtual medicaldevice based at least in part on the determined emplacement of thesecond virtual medical device.

Clause 85. The non-transitory computer-readable medium of clauses 82-84,wherein the computer executable instructions further cause the one ormore processors to cause the display to concurrently display aperspective view of:

a first trajectory indicator corresponding to the trajectory of thefirst limb member along the drive path, or

a second trajectory indicator corresponding to the trajectory of thesecond limb member along the drive path.

Clause 86. The non-transitory computer-readable medium of any of clauses82-85, wherein the drive path is parallel to at least one of the jointmember axis, the first axis, or the second axis.

Clause 87. The non-transitory computer-readable medium of any of clauses82-86, wherein the drive path is perpendicular to at least one of thejoint member axis, the first axis, or the second axis.

Clause 88. The non-transitory computer-readable medium of any of clauses82-87, wherein the drive path is predefined.

Clause 89. The non-transitory computer-readable medium of any of clauses82-88, wherein the drive path is dynamic and is based at least in parton detected movement of first medical device or the second medicaldevice.

Clause 90. The non-transitory computer-readable medium of any of clauses82-89, wherein a primary axis of the first medical device is the jointmember axis.

Clause 91. The non-transitory computer-readable medium of any of clauses82-90, wherein a secondary axis of the first medical device is the firstaxis and/or the second axis.

Clause 92. The non-transitory computer-readable medium of any of clauses82-91, wherein the first axis is parallel to the joint member axis.

Clause 93. The non-transitory computer-readable medium of any of clauses82-92, wherein the first axis is different from the joint member axis.

Clause 94. The non-transitory computer-readable medium of any of clauses82-93, wherein the first intersection indicator includes an outline of across-section of the first limb member.

Clause 95. The non-transitory computer-readable medium of any of clauses82-94, wherein the second intersection indicator includes an outline ofa cross-section of the second limb member.

Clause 96. The non-transitory computer-readable medium of any of clauses82-95, wherein the first intersection indicator and/or the secondintersection indicator includes a cross-sectional outline of a knifepath and/or the first virtual medical device, wherein the knife pathcorresponds to a transecting knife of the first medical device.

Clause 97. The non-transitory computer-readable medium of any of clauses82-96, wherein the first intersection and/or the second intersectionindicator includes an indication of at least a portion of the imageplane.

Clause 98. The non-transitory computer-readable medium of clause 97,wherein the indication of at least a portion of the image planecomprises a box representative of a portion of the image plane.

Clause 99. The non-transitory computer-readable medium of clause 85,wherein the first trajectory indicator extends to at least a distalportion of the first limb member and corresponds to an axis along thedrive path that is parallel to the joint member axis.

Clause 100. The non-transitory computer-readable medium of clause 85,wherein the first trajectory indicator corresponds to an axis thatintersects with the first axis at a distal portion of the first limbmember and is parallel to the joint member axis and the drive path.

Clause 101. The non-transitory computer-readable medium of clause 85,wherein the second trajectory indicator extends to at least a distalportion of the second limb member and corresponds to an axis along thedrive path that is different from and parallel to the joint member axis.

Clause 102. The non-transitory computer-readable medium of clause 85,wherein the second trajectory indicator corresponds to an axis thatintersects with the second axis at a distal portion of the second limbmember and is parallel to the joint member axis and the drive path.

Clause 103. The non-transitory computer-readable medium of any ofclauses 82-102, wherein the computer system is further configured todetermine a cutting path associated with the first medical device andcause the display to concurrently display a perspective view of thecutting path.

Clause 104. The non-transitory computer-readable medium of clause 103,wherein the cutting path corresponds to a trajectory of a knife of thefirst medical device.

Clause 105. The non-transitory computer-readable medium of any ofclauses 82-104, wherein the computer executable instructions furthercause the one or more processors to:

determine an emplacement of an affected region based at least in part onthe determined emplacement of the first virtual medical device; and

causing the display to concurrently display a perspective view of:

-   -   a 3D rendering of at least a portion of the affected region        based at least in part on the determined emplacement of the        virtual medical device, or    -   an affected region intersection indicator indicating an        intersection between a trajectory of the affected region along        the drive path.

Clause 106. The non-transitory computer-readable medium of clause 105,wherein the affected region corresponds to a volume located between thefirst limb member and the second limb member.

Clause 107. The non-transitory computer-readable medium of clause 105,wherein the at least a portion of the affected region comprises astapling path of at least one staple of the first medical device.

Clause 108. The non-transitory computer-readable medium of clause 105,wherein the at least a portion of the affected region comprises acutting path of a knife of the first medical device.

Clause 109. The non-transitory computer-readable medium of clause 105,wherein the at least a portion of the affected region comprises theintersection between the affected region and the medical image.

Clause 110. The non-transitory computer-readable medium of any ofclauses 82-109, wherein the computer executable instructions furthercause the one or more processors to determine an intersection betweenthe affected region and the medical image.

Clause 111. The non-transitory computer-readable medium of any ofclauses 82-110, wherein the computer executable instructions furthercause the one or more processors to receive operating parameterscorresponding to the first medical device and determine the emplacementof an affected region based at least in part on the received operatingparameters.

Clause 112. The non-transitory computer-readable medium of any ofclauses 82-110, wherein the computer executable instructions furthercause the one or more processors to:

determine an emplacement of a plane-of-interest based at least in parton the determined emplacement of one of the medical image;

determine an emplacement of a ghost medical device based at least inpart on the determined emplacement of the first virtual medical deviceand the determined emplacement of the plane-of-interest; and

cause the display to concurrently display a perspective view of a 3Drendering of at least a portion of the ghost medical device on theplane-of-interest based at least in part on the determined emplacementof the ghost medical device.

Clause 113. The non-transitory computer-readable medium of any ofclauses 82-110, wherein the computer executable instructions furthercause the one or more processors to:

determine an emplacement of a plane-of-interest based at least in parton the determined emplacement of one of the first virtual medicaldevice;

determine an emplacement of a ghost medical image based at least in parton the determined emplacement of the medical image and the determinedemplacement of the plane-of-interest; and

cause the display to currently display a perspective view of a 3Drendering of at least a portion of the ghost medical image on theplane-of-interest based at least in part on the determined emplacementof the ghost medical image.

Clause 114. The non-transitory computer-readable medium of any one ofclause 112 or 73, wherein the computer executable instructions furthercause the one or more processors to project the display object on theplane-of-interest to display the ghost display object.

Clause 115. The non-transitory computer-readable medium of Clause 114,wherein the computer executable instructions further cause the one ormore processors to determine at least one of portions, points, orcoordinates of the plane-of-interest that are closest to the displayobject.

Clause 116. The non-transitory computer-readable medium of any ofclauses 114 or 115, wherein the computer executable instructions furthercause the one or more processors to determine an arc path correspondingto the display object and the plane-of-interest.

Clause 117. The non-transitory computer-readable medium of any ofclauses 114-116, wherein the computer executable instructions furthercause the one or more processors to determine which of a plurality ofconnecting lines intersect the display object, wherein the plurality ofconnecting lines extend orthogonally from the plane-of-interest to thedisplay object.

Clause 118. The non-transitory computer-readable medium of any ofclauses 114-117, wherein the computer executable instructions furthercause the one or more processors to use a virtual light source that isat least as large as the display object to be projected, wherein thelight source is distal the plane-of-interest relative to the displayobject.

Clause 119. The non-transitory computer-readable medium of any ofclauses 114-118, wherein the ghost display object includes across-section and/or outline of display object.

Clause 120. The non-transitory computer-readable medium of any ofclauses 114-119, wherein the ghost display object includes across-section and/or outline of display object.

Clause 121. The method of any of the Clauses 3-40, wherein the receivedfirst emplacement data and the received second emplacement datacorrespond to a first position sensor.

Clause 122. The system of any of clauses 43-80, wherein the receivedfirst emplacement data and the received second emplacement datacorrespond to a first position sensor.

Clause 123. The non-transitory computer-readable medium of any ofclauses 83-120, wherein the received first emplacement data and thereceived second emplacement data correspond to a first position sensor.

Various example embodiments of methods, systems and non-transitorycomputer-readable medium relating to projecting a display object onto aplane-of-interest to display a ghost display object can be found in thefollowing clauses:

Clause 1. A method for medical instrument navigation, the methodcomprising: receiving first emplacement data associated with a firstmedical device;

receiving second emplacement data associated with a second medicaldevice;

receiving a medical image corresponding to the second medical device;

determining an emplacement of a first display object corresponding tothe first medical device based at least in part on the received firstemplacement data;

determining an emplacement of a second display object corresponding tothe second medical device based at least in part on the received secondemplacement data;

determining an emplacement of the medical image based at least in parton the received second emplacement data;

determining an emplacement of a plane-of-interest based at least in parton the emplacement of one of the first display object or the medicalimage;

determining an emplacement of a ghost display object based at least inpart on the determined emplacement of the other of the first displayobject or the medical image display object and the

determined emplacement of the plane-of-interest; and

causing one or more displays to concurrently display a perspective viewof:

-   -   a 3D rendering of at least a portion of the first display object        based at least in part on the determined emplacement of the        first display object,    -   a 3D rendering of at least a portion of the second display        object based at least in part on the determined emplacement of        the second display object,    -   a 3D rendering of at least a portion of the medical image based        at least in part on the determined emplacement of the medical        image,    -   a 3D rendering of at least a portion of the ghost display object        on the plane-of-interest based at least in part on the        determined emplacement of the ghost display object.

Clause 2. A method for medical instrument navigation, the methodcomprising:

determining an emplacement of a display object;

determining an emplacement of a plane-of-interest;

determining an emplacement of a ghost display object based at least inpart on the determined emplacement of the display object and thedetermined emplacement the plane-of-interest; and

causing one or more displays to display concurrently display:

-   -   a 3D rendering of at least a portion of the display object, and    -   a 3D rendering of at least a portion of the ghost display object        on the plane-of-interest.

Clause 3. The method of Clause 2, further comprising:

receiving first emplacement data associated with a first medical device;

receiving second emplacement data associated with a second medicaldevice;

wherein said determining the emplacement of the display object is basedat least in part on the received first emplacement data and saiddetermining the emplacement of the plane-of-interest is based at leastin part on the received second emplacement data.

Clause 4. The method of Clause 3, further comprising

receiving a medical image corresponding to the second medical device;and

determining an emplacement of the medical image based at least in parton the received second emplacement data,

wherein said determining the emplacement of the plane-of-interest isbased at least in part on the determined emplacement of the medicalimage.

Clause 5. The method of Clause 3, further comprising

receiving a medical image corresponding to the first medical device; and

determining an emplacement of the medical image based at least in parton the received first emplacement data,

wherein said determining the emplacement of the display object is basedat least in part on the determined emplacement of the medical image.

Clause 6. The method of any of clauses 2-5, further comprisingprojecting the display object on the plane-of-interest to display theghost display object.

Clause 7. The method of any of clauses 6-6, wherein said projectingcomprises determining at least one of portions, points, or coordinatesof the plane-of-interest that are closest to the display object.

Clause 8. The method of any of clauses 6-7, wherein said projectingcomprises determining an arc path corresponding to the display objectand the plane-of-interest.

Clause 9. The method of any of clauses 6-8, wherein said projectingcomprises determining which of a plurality of connecting lines intersectthe display object, wherein the plurality of connecting lines extendorthogonally from the plane-of-interest to the display object.

Clause 10. The method of any of clauses 6-9, wherein said projectingcomprises using a virtual light source that is at least as large as thedisplay object to be projected, wherein the virtual light source isdistal to the plane-of-interest relative to the display object.

Clause 11. The method of any of clauses 2-10, wherein the ghost displayobject includes a cross-section and/or outline of display object.

Clause 12. The method of any of clauses 2-11, wherein the ghost displayobject corresponds to at least one of a virtual medical device, amedical image, or an affected region.

Clause 13. The method of any of clauses 2-12, wherein theplane-of-interest corresponds to at least one of a medical device planeor an image plane.

Clause 14. The method of any of clauses 2-13, wherein the display objectis an affected region and wherein the affected region corresponds to avolume located between a first limb member and a second limb member of amedical device.

Clause 15. The method of any of clauses 2-14, wherein the display objectis an affected region and wherein the affected region comprises astapling path of at least one staple of a medical device.

Clause 16. The method of any of clauses 2-15, wherein the display objectis an affected region and wherein the affected region comprises acutting path of a knife of a medical device.

Clause 17. A system comprising:

a computer system in communication with a display, the computer systemcomprising a computer processor and a non-transitory storage medium,wherein the computer system is configured to:

receive first emplacement data associated with a first medical device,

receive second emplacement data associated with a second medical device,

receive a medical image corresponding to the second medical device,

determine an emplacement of a first display object corresponding to thefirst medical device based at least in part on the received firstemplacement data,

determine an emplacement of a second display object corresponding to thesecond medical device based at least in part on the received secondemplacement data,

determine an emplacement of the medical image based at least in part onthe received second emplacement data,

determine an emplacement of a plane-of-interest based at least in parton the emplacement of one of the first display object or the medicalimage,

determine an emplacement of a ghost display object based at least inpart on the determined emplacement of the other of the first displayobject or the medical image display object and the determinedemplacement of the plane-of-interest, and

cause a display to concurrently display a perspective view of:

-   -   a 3D rendering of at least a portion of the first display object        based at least in part on the determined emplacement of the        first display object,    -   a 3D rendering of at least a portion of the second display        object based at least in part on the determined emplacement of        the second display object,    -   a 3D rendering of at least a portion of the medical image based        at least in part on the determined emplacement of the medical        image,    -   a 3D rendering of at least a portion of the ghost display object        on the plane-of-interest based at least in part on the        determined emplacement of the ghost display object; and

the display.

Clause 18. A system comprising:

a computer system in communication with a display, the computer systemcomprising a computer processor and a non-transitory storage medium,wherein the computer system is configured to:

determine an emplacement of a display object,

determine an emplacement of a plane-of-interest,

determine an emplacement of a ghost display object based at least inpart on the determined emplacement of the display object and thedetermined emplacement the plane-of-interest, and

cause one or more displays to display concurrently display:

-   -   a 3D rendering of at least a portion of the display object, and    -   a 3D rendering of at least a portion of the ghost display object        on the plane-of-interest; and        the display.

Clause 19. The system of Clause 18, wherein the computer system isfurther configured to:

receive first emplacement data associated with a first medical device;

receiving second emplacement data associated with a second medicaldevice;

wherein said determining the emplacement of the display object is basedat least in part on the received first emplacement data and saiddetermining the emplacement of the plane-of-interest is based at leastin part on the received second emplacement data.

The system of Clause 19, wherein the computer system is furtherconfigured to:

receive a medical image corresponding to the second medical device;

determine an emplacement of the medical image based at least in part onthe received second emplacement data; and

determine the emplacement of the plane-of-interest based at least inpart on the determined emplacement of the medical image.

Clause 20. The system of Clause 19, wherein the computer system isfurther configured to:

receive a medical image corresponding to the first medical device;

determine an emplacement of the medical image based at least in part onthe received first emplacement data; and

determine the emplacement of the display object based at least in parton the determined emplacement of the medical image.

Clause 21. The system of any of clauses 18-21, wherein the computersystem is further configured to project the display object on theplane-of-interest to display the ghost display object.

Clause 22. The system of any of clauses 21-22, wherein the computersystem is further configured to determine at least one of portions,points, or coordinates of the plane-of-interest that are closest to thedisplay object.

Clause 23. The system of any of clauses 21-23, wherein the computersystem is further configured to determine an arc path corresponding tothe display object and the plane-of-interest.

Clause 24. The system of any of clauses 21-24, wherein the computersystem is further configured to determine which of a plurality ofconnecting lines intersect the display object, wherein the plurality ofconnecting lines extend orthogonally from the plane-of-interest to thedisplay object.

Clause 25. The system of any of clauses 21-25, wherein the computersystem is further configured to use a virtual light source that is atleast as large as the display object to be projected, wherein thevirtual light source is distal to the plane-of-interest relative to thedisplay object.

Clause 26. The system of any of clauses 18-26, wherein the ghost displayobject includes a cross-section and/or outline of display object.

Clause 27. The system of any of clauses 18-27, wherein the ghost displayobject corresponds to at least one of a virtual medical device, amedical image, or an affected region.

Clause 28. The system of any of clauses 18-28, wherein theplane-of-interest corresponds to at least one of a medical device planeor an image plane.

Clause 29. The system of any of clauses 18-29, wherein the displayobject is an affected region and wherein the affected region correspondsto a volume located between a first limb member and a second limb memberof a medical device.

Clause 30. The system of any of clauses 18-30, wherein the displayobject is an affected region and wherein the affected region comprises astapling path of at least one staple of a medical device.

Clause 31. The system of any of clauses 18-31, wherein the displayobject is an affected region and wherein the affected region comprises acutting path of a knife of a medical device.

Clause 32. A non-transitory computer-readable medium comprising computerexecutable instructions that when executed by one or more processorscause the one or more processors to:

receive first emplacement data associated with a first medical device,

receive second emplacement data associated with a second medical device,

receive a medical image corresponding to the second medical device,

determine an emplacement of a first display object corresponding to thefirst medical device based at least in part on the received firstemplacement data,

determine an emplacement of a second display object corresponding to thesecond medical device based at least in part on the received secondemplacement data,

determine an emplacement of the medical image based at least in part onthe received second emplacement data,

determine an emplacement of a plane-of-interest based at least in parton the emplacement of one of the first display object or the medicalimage,

determine an emplacement of a ghost display object based at least inpart on the determined emplacement of the other of the first displayobject or the medical image display object and the determinedemplacement of the plane-of-interest, and

cause a display to concurrently display a perspective view of:

-   -   a 3D rendering of at least a portion of the first display object        based at least in part on the determined emplacement of the        first display object,    -   a 3D rendering of at least a portion of the second display        object based at least in part on the determined emplacement of        the second display object,    -   a 3D rendering of at least a portion of the medical image based        at least in part on the determined emplacement of the medical        image,    -   a 3D rendering of at least a portion of the ghost display object        on the plane-of-interest based at least in part on the        determined emplacement of the ghost display object.

Clause 33. A non-transitory computer-readable medium comprising computerexecutable instructions that when executed by one or more processorscause the one or more processors to:

determine an emplacement of a display object,

determine an emplacement of a plane-of-interest,

determine an emplacement of a ghost display object based at least inpart on the determined emplacement of the display object and thedetermined emplacement the plane-of-interest, and

cause one or more displays to display concurrently display:

-   -   a 3D rendering of at least a portion of the display object, and    -   a 3D rendering of at least a portion of the ghost display object        on the plane-of-interest.

Clause 34. The non-transitory computer-readable medium of Clause 18,wherein the computer executable instructions further cause the one ormore processors to:

receive first emplacement data associated with a first medical device;

receiving second emplacement data associated with a second medicaldevice;

wherein said determining the emplacement of the display object is basedat least in part on the received first emplacement data and saiddetermining the emplacement of the plane-of-interest is based at leastin part on the received second emplacement data.

Clause 35. The non-transitory computer-readable medium of Clause 19,wherein the computer executable instructions further cause the one ormore processors to:

receive a medical image corresponding to the second medical device;

determine an emplacement of the medical image based at least in part onthe received second emplacement data; and

determine the emplacement of the plane-of-interest based at least inpart on the determined emplacement of the medical image.

Clause 36. The non-transitory computer-readable medium of Clause 19,wherein the computer executable instructions further cause the one ormore processors to:

receive a medical image corresponding to the first medical device;

determine an emplacement of the medical image based at least in part onthe received first emplacement data; and

determine the emplacement of the display object based at least in parton the determined emplacement of the medical image.

Clause 37. The non-transitory computer-readable medium of any of clauses18-21, wherein the computer executable instructions further cause theone or more processors to project the display object on theplane-of-interest to display the ghost display object.

Clause 38. The non-transitory computer-readable medium of any of clauses21-22, wherein the computer executable instructions further cause theone or more processors to determine at least one of portions, points, orcoordinates of the plane-of-interest that are closest to the displayobject.

Clause 39. The non-transitory computer-readable medium of any of clauses21-23, wherein the computer executable instructions further cause theone or more processors to determine an arc path corresponding to thedisplay object and the plane-of-interest.

Clause 40. The non-transitory computer-readable medium of any of clauses21-24, wherein the computer executable instructions further cause theone or more processors to determine which of a plurality of connectinglines intersect the display object, wherein the plurality of connectinglines extend orthogonally from the plane-of-interest to the displayobject.

Clause 41. The non-transitory computer-readable medium of any of clauses21-25, wherein the computer executable instructions further cause theone or more processors to use a virtual light source that is at least aslarge as the display object to be projected, wherein the virtual lightsource is distal to the plane-of-interest relative to the displayobject.

Clause 42. The non-transitory computer-readable medium of any of clauses18-26, wherein the ghost display object includes a cross-section and/oroutline of display object.

Clause 43. The non-transitory computer-readable medium of any of clauses18-27, wherein the ghost display object corresponds to at least one of avirtual medical device, a medical image, or an affected region.

Clause 44. The non-transitory computer-readable medium of any of clauses18-28, wherein the plane-of-interest corresponds to at least one of amedical device plane or an image plane.

Clause 45. The non-transitory computer-readable medium of any of clauses18-29, wherein the display object is an affected region and wherein theaffected region corresponds to a volume located between a first limbmember and a second limb member of a medical device.

Clause 46. The non-transitory computer-readable medium of any of clauses18-30, wherein the display object is an affected region and wherein theaffected region comprises a stapling path of at least one staple of amedical device.

Clause 47. The non-transitory computer-readable medium of any of clauses18-31, wherein the display object is an affected region and wherein theaffected region comprises a cutting path of a knife of a medical device.

Various example embodiments of methods, systems and non-transitorycomputer-readable medium relating to displaying an affected region of amedical device can be found in the following clauses:

Clause 1. A method for medical instrument navigation, the methodcomprising:

receiving emplacement data associated with a medical device, the medicaldevice comprising a joint member having a joint member axis, a firstlimb member having a first axis, and a second limb member having asecond axis that is different from the joint member axis and the firstaxis;

receiving operating parameters corresponding to the medical device;

determining an emplacement of a virtual medical device corresponding tothe medical device based at least in part on the received emplacementdata;

determining an emplacement of an affected region based at least in parton the received emplacement data and the operating parameters, whereinthe affected region corresponds to a volume located between the firstlimb member and the second limb member; and

causing one or more displays to concurrently display a perspective viewof:

-   -   a 3D rendering of at least a portion of the virtual medical        device based at least in part on the determined emplacement of        the virtual medical device, the 3D rendering of the virtual        medical device including at least the first limb member and the        second limb member, and    -   a 3D rendering of at least a portion of the affected region        based at least in part on the determined emplacement of the        affected region, or    -   an intersection indicator indicating an intersection between an        image plane and a trajectory of the affected region along a        drive path that is parallel to the joint member axis.

Clause 2. A method for medical instrument navigation, the methodcomprising:

determining an emplacement of a virtual medical device corresponding toa medical device, the medical device comprising a joint member having ajoint member axis, a first limb member having a first axis, and a secondlimb member having a second axis that is different from the joint memberaxis and the first axis;

determining an emplacement of an affected region based at least in parton the determined emplacement of the virtual medical device; and

causing one or more displays to display at least one of:

-   -   a 3D rendering of at least a portion of the affected region        based at least in part on the determined emplacement of the        virtual medical device, or    -   an intersection indicator indicating an intersection between an        image plane and a trajectory of the affected region along a        drive path that is parallel to the joint member axis.

Clause 3. The method of Clause 2, further comprising:

receiving emplacement data associated with the medical device; and

receiving operating parameters corresponding to the medical device,

wherein said determining the emplacement of the virtual medical deviceis based at least in part on the received emplacement data, wherein saiddetermining the emplacement of the affected region is based at least inpart on the received emplacement data and the operating parameters.

Clause 4. The method of Clause 2, wherein the 3D rendering of thevirtual medical includes at least the first limb member and the secondlimb member.

Clause 5. The method of Clause 2, further comprising causing the one ormore displays to display a 3D rendering of at least a portion of thevirtual medical device based at least in part on the determinedemplacement of the virtual medical device, wherein the 3D rendering ofthe virtual medical device includes at least a first virtual limb membercorresponding to the first limb member and a second virtual limb membercorresponding to the second limb member.

Clause 6. The method of Clause 2, wherein the affected regioncorresponds to a volume located between the first limb member and thesecond limb member.

Clause 7. The method of Clause 2, wherein the affected region comprisesa stapling path of at least one staple of the medical device.

Clause 8. The method of Clause 2, wherein the affected region comprisesa cutting path of a knife of the medical device.

Clause 9. A system comprising:

a computer system in communication with a display, the computer systemcomprising a computer processor and a non-transitory storage medium,wherein the computer system is configured to:

-   -   receive emplacement data associated with a medical device, the        medical device comprising a joint member having a joint member        axis, a first limb member having a first axis, and a second limb        member having a second axis that is different from the joint        member axis and the first axis;    -   receive operating parameters corresponding to the medical        device,    -   determine an emplacement of a virtual medical device        corresponding to the medical device based at least in part on        the received emplacement data,    -   determine an emplacement of an affected region based at least in        part on the received emplacement data and the operating        parameters, wherein the affected region corresponds to a volume        located between the first limb member and the second limb        member, and    -   cause a display to concurrently display a perspective view of:        -   a 3D rendering of at least a portion of the virtual medical            device based at least in part on the determined emplacement            of the virtual medical device, the 3D rendering of the            virtual medical device including at least the first limb            member and the second limb member, and        -   a 3D rendering of at least a portion of the affected region            based at least in part on the determined emplacement of the            affected region, or        -   an intersection indicator indicating an intersection between            an image plane and a trajectory of the affected region along            a drive path that is parallel to the joint member axis; and

the display.

Clause 10. A system comprising:

a computer system in communication with a display, the computer systemcomprising a computer processor and a non-transitory storage medium,wherein the computer system is configured to:

-   -   determine an emplacement of a virtual medical device        corresponding to a medical device, the medical device comprising        a joint member having a joint member axis, a first limb member        having a first axis, and a second limb member having a second        axis that is different from the joint member axis and the first        axis,    -   determine an emplacement of an affected region based at least in        part on the determined emplacement of the virtual medical        device, and    -   cause a display to display at least one of:        -   a 3D rendering of at least a portion of the affected region            based at least in part on the determined emplacement of the            virtual medical device, or        -   an intersection indicator indicating an intersection between            an image plane and a trajectory of the affected region along            a drive path that is parallel to the joint member axis; and

the display.

Clause 11. The system of Clause 10, wherein the computer system isfurther configured to:

receive emplacement data associated with the medical device;

receive operating parameters corresponding to the medical device;

determine the emplacement of the virtual medical device based at leastin part on the received emplacement data; and

determine the emplacement of the affected region based at least in parton the received emplacement data and the operating parameters.

Clause 12. The system of any of clauses 10-11, wherein the 3D renderingof the virtual medical includes at least the first limb member and thesecond limb member.

Clause 13. The system of any of clauses 10-12, wherein the computersystem is further configured to cause the display to display a 3Drendering of at least a portion of the virtual medical device based atleast in part on the determined emplacement of the virtual medicaldevice, wherein the 3D rendering of the virtual medical device includesat least a first virtual limb member corresponding to the first limbmember and a second virtual limb member corresponding to the second limbmember.

Clause 14. The system of any of clauses 10-13, wherein the affectedregion corresponds to a volume located between the first limb member andthe second limb member.

Clause 15. The system of any of clauses 10-14, wherein the affectedregion comprises a stapling path of at least one staple of the medicaldevice.

Clause 16. The system of any of clauses 10-15, wherein the affectedregion comprises a cutting path of a knife of the medical device.

Clause 17. A non-transitory computer-readable medium comprising computerexecutable instructions that when executed by one or more processorscause the one or more processors to:

-   -   receive emplacement data associated with a medical device, the        medical device comprising a joint member having a joint member        axis, a first limb member having a first axis, and a second limb        member having a second axis that is different from the joint        member axis and the first axis;    -   receive operating parameters corresponding to the medical        device,    -   determine an emplacement of a virtual medical device        corresponding to the medical device based at least in part on        the received emplacement data,    -   determine an emplacement of an affected region based at least in        part on the received emplacement data and the operating        parameters, wherein the affected region corresponds to a volume        located between the first limb member and the second limb        member, and    -   cause a display to concurrently display a perspective view of:        -   a 3D rendering of at least a portion of the virtual medical            device based at least in part on the determined emplacement            of the virtual medical device, the 3D rendering of the            virtual medical device including at least the first limb            member and the second limb member, and        -   a 3D rendering of at least a portion of the affected region            based at least in part on the determined emplacement of the            affected region, or        -   an intersection indicator indicating an intersection between            an image plane and a trajectory of the affected region along            a drive path that is parallel to the joint member axis.

Clause 18. A non-transitory computer-readable medium comprising computerexecutable instructions that when executed by one or more processorscause the one or more processors to:

-   -   determine an emplacement of a virtual medical device        corresponding to a medical device, the medical device comprising        a joint member having a joint member axis, a first limb member        having a first axis, and a second limb member having a second        axis that is    -   different from the joint member axis and the first axis,

determine an emplacement of an affected region based at least in part onthe determined emplacement of the virtual medical device, and

-   -   cause a display to display at least one of:        -   a 3D rendering of at least a portion of the affected region            based at least in part on the determined emplacement of the            virtual medical device, or        -   an intersection indicator indicating an intersection between            an image plane and a trajectory of the affected region along            a drive path that is parallel to the joint member axis.

Clause 19. The non-transitory computer-readable medium of Clause 18,wherein the computer executable instructions further cause the one ormore processors to:

receive emplacement data associated with the medical device;

receive operating parameters corresponding to the medical device;

determine the emplacement of the virtual medical device based at leastin part on the received emplacement data; and

determine the emplacement of the affected region based at least in parton the received emplacement data and the operating parameters.

Clause 20. The non-transitory computer-readable medium of any of clauses18-19, wherein the 3D rendering of the virtual medical includes at leastthe first limb member and the second limb member.

Clause 21. The non-transitory computer-readable medium of any of clauses18-20, wherein the computer executable instructions further cause theone or more processors to cause the display to display a 3D rendering ofat least a portion of the virtual medical device based at least in parton the determined emplacement of the virtual medical device, wherein the3D rendering of the virtual medical device includes at least a firstvirtual limb member corresponding to the first limb member and a secondvirtual limb member corresponding to the second limb member.

Clause 22. The non-transitory computer-readable medium of any of clauses18-21, wherein the affected region corresponds to a volume locatedbetween the first limb member and the second limb member.

Clause 23. The non-transitory computer-readable medium of any of clauses18-22, wherein the affected region comprises a stapling path of at leastone staple of the medical device.

Clause 24. The non-transitory computer-readable medium of any of clauses18-23, wherein the affected region comprises a cutting path of a knifeof the medical device.

Various example embodiments of methods, systems and non-transitorycomputer-readable medium relating to determining emplacement of avirtual medical device and a medical image based at least in part onreceived emplacement data associated with a medical device can be foundin the following clauses:

Clause 1. A method for medical instrument navigation, the methodcomprising: receiving emplacement data associated with a first medicaldevice, the first medical device comprising a joint member having ajoint member axis, a first limb member having a first axis, and a secondlimb member having a second axis that is different from the joint memberaxis and the first axis;

determining an emplacement of a first virtual medical devicecorresponding to the first medical device based at least in part on thereceived emplacement data;

determining an emplacement of a second virtual medical devicecorresponding to a second medical device based at least in part on thereceived emplacement data;

determining an emplacement of a medical image based at least in part onthe received emplacement data; and

causing one or more displays to concurrently display a perspective viewof:

-   -   a 3D rendering of at least a portion of the first virtual        medical device based at least in part on the determined        emplacement of the first virtual medical device, the 3D        rendering of the first virtual medical device including at least        the first limb member and the second limb member,    -   a 3D rendering of at least a portion of the second virtual        medical device based at least in part on the determined        emplacement of the second virtual medical device, and    -   a 3D rendering of at least a portion of the medical image based        at least in part on the determined emplacement of the medical        image.

Clause 2. A method for medical instrument navigation, the methodcomprising:

-   -   determining an emplacement of a display object based at least in        part on first emplacement data;

determining an emplacement of a medical image based at least in part onthe first emplacement data; and

causing one or more displays to concurrently display:

-   -   a 3D rendering of at least a portion of the display object, and    -   a 3D rendering of at least a portion of the medical image.

Clause 3. The method of Clause 2, wherein the display object is a firstvirtual medical device corresponding to a first medical devicecomprising a joint member having a joint member axis, a first limbmember having a first axis, and a second limb member having a secondaxis that is different from the joint member axis and the first axis.

Clause 4. The method of Clause 3, further comprising:

receiving the first emplacement data, the first emplacement dataassociated with the first medical device; and

determining an emplacement of a second virtual medical devicecorresponding to a second medical device based at least in part on thereceived first emplacement data.

Clause 5. The method of any of clauses 2-4, wherein the display objectis a first virtual medical device and wherein causing the one or moredisplays to concurrently display further comprises causing concurrentdisplay a 3D rendering of at least a portion of the first virtualmedical device.

Clause 6. The method of Clause 5, the 3D rendering of at least a portionof the first virtual medical device includes at least a first limbmember and a second limb member of the virtual medical device.

Clause 7. The method of any of clauses 2-6, where causing one or moredisplays to concurrently display further comprises causing the one ormore displays to display:

a 3D rendering of at least a portion of the second virtual medicaldevice based at least in part on the determined emplacement of thesecond virtual medical device; and

a 3D rendering of at least a portion of the medical image based at leastin part on the determined emplacement of the medical image.

Clause 8. A system comprising:

a computer system in communication with a display, the computer systemcomprising a computer processor and a non-transitory storage medium,wherein the computer system is configured to:

-   -   receive emplacement data associated with a first medical device,        the first medical device comprising a joint member having a        joint member axis, a first limb member having a first axis, and        a second limb member having a second axis that is different from        the joint member axis and the first axis,    -   determine an emplacement of a first virtual medical device        corresponding to the first medical device based at least in part        on the received emplacement data,    -   determine an emplacement of a second virtual medical device        corresponding to a second medical device based at least in part        on the received emplacement data,    -   determine an emplacement of a medical image based at least in        part on the received emplacement data, and    -   cause a display to concurrently display a perspective view of:        -   a 3D rendering of at least a portion of the first virtual            medical device based at least in part on the determined            emplacement of the first virtual medical device, the 3D            rendering of the first virtual medical device including at            least the first limb member and the second limb member,        -   a 3D rendering of at least a portion of the second virtual            medical device based at least in part on the determined            emplacement of the second virtual medical device, and        -   a 3D rendering of at least a portion of the medical image            based at least in part on the determined emplacement of the            medical image; and

the display.

Clause 9. A system comprising:

a computer system in communication with a display, the computer systemcomprising a computer processor and a non-transitory storage medium,wherein the computer system is configured to:

-   -   determine an emplacement of a display object based at least in        part on first emplacement data,    -   determine an emplacement of a medical image based at least in        part on the first emplacement data, and    -   cause one or more displays to concurrently display:        -   a 3D rendering of at least a portion of the display object,            and        -   a 3D rendering of at least a portion of the medical image;            and

the display.

Clause 10. The system of Clause 9, wherein the display object is a firstvirtual medical device corresponding to a first medical devicecomprising a joint member having a joint member axis, a first limbmember having a first axis, and a second limb member having a secondaxis that is different from the joint member axis and the first axis.

Clause 11. The system of Clause 10, wherein the computer system isfurther configured to:

receive the first emplacement data, the first emplacement dataassociated with the first medical device; and

determine an emplacement of a second virtual medical devicecorresponding to a second medical device based at least in part on thereceived first emplacement data.

Clause 12. The system of any of clauses 9-11, wherein the display objectis a first virtual medical device and the computer system is furtherconfigured to cause the display to concurrently display a 3D renderingof at least a portion of the first virtual medical device.

Clause 13. The system of Clause 12, the 3D rendering of at least aportion of the first virtual medical device includes at least a firstlimb member and a second limb member of the virtual medical device.

Clause 14. The system of any of clauses 9-14, where the computer systemis further configured to cause the display to concurrently display:

a 3D rendering of at least a portion of the second virtual medicaldevice based at least in part on the determined emplacement of thesecond virtual medical device; and

a 3D rendering of at least a portion of the medical image based at leastin part on the determined emplacement of the medical image.

Clause 15. A non-transitory computer-readable medium comprising computerexecutable instructions that when executed by one or more processorscause the one or more processors to:

-   -   receive emplacement data associated with a first medical device,        the first medical device comprising a joint member having a        joint member axis, a first limb member having a first axis, and        a second limb member having a second axis that is different from        the joint member axis and the first axis,    -   determine an emplacement of a first virtual medical device        corresponding to the first medical device based at least in part        on the received emplacement data,    -   determine an emplacement of a second virtual medical device        corresponding to a second medical device based at least in part        on the received emplacement data,    -   determine an emplacement of a medical image based at least in        part on the received emplacement data, and    -   cause a display to concurrently display a perspective view of:        -   a 3D rendering of at least a portion of the first virtual            medical device based at least in part on the determined            emplacement of the first virtual medical device, the 3D            rendering of the first virtual medical device including at            least the first limb member and the second limb member,        -   a 3D rendering of at least a portion of the second virtual            medical device based at least in part on the determined            emplacement of the second virtual medical device, and        -   a 3D rendering of at least a portion of the medical image            based at least in part on the determined emplacement of the            medical image.

Clause 16. A non-transitory computer-readable medium comprising computerexecutable instructions that when executed by one or more processorscause the one or more processors to:

-   -   determine an emplacement of a display object based at least in        part on first emplacement data,    -   determine an emplacement of a medical image based at least in        part on the first emplacement data, and    -   cause one or more displays to concurrently display:        -   a 3D rendering of at least a portion of the display object,            and        -   a 3D rendering of at least a portion of the medical image.

Clause 17. The system of Clause 16, wherein the display object is afirst virtual medical device corresponding to a first medical devicecomprising a joint member having a joint member axis, a first limbmember having a first axis, and a second limb member having a secondaxis that is different from the joint member axis and the first axis.

Clause 18. The non-transitory computer-readable medium of Clause 17,wherein the computer executable instructions further cause the one ormore processors to:

receive the first emplacement data, the first emplacement dataassociated with the first medical device; and

determine an emplacement of a second virtual medical devicecorresponding to a second medical device based at least in part on thereceived first emplacement data.

Clause 19. The non-transitory computer-readable medium of any of clauses16-18, wherein the display object is a first virtual medical device andwherein the computer executable instructions further cause the one ormore processors to cause the display to concurrently display a 3Drendering of at least a portion of the first virtual medical device.

Clause 20. The non-transitory computer-readable medium of Clause 19, the3D rendering of at least a portion of the first virtual medical deviceincludes at least a first limb member and a second limb member of thevirtual medical device.

Clause 21. The non-transitory computer-readable medium of any of clauses16-20, wherein the computer executable instructions further cause theone or more processors to cause the display to concurrently display:

a 3D rendering of at least a portion of the second virtual medicaldevice based at least in part on the determined emplacement of thesecond virtual medical device; and

a 3D rendering of at least a portion of the medical image based at leastin part on the determined emplacement of the medical image.

Terminology

Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements, and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” As used herein, the terms “connected,”“coupled,” or any variant thereof means any connection or coupling,either direct or indirect, between two or more elements; the coupling orconnection between the elements can be physical, logical, or acombination thereof. Additionally, the words “herein,” “above,” “below,”and words of similar import, when used in this application, refer tothis application as a whole and not to any particular portions of thisapplication. Where the context permits, words in the above DetailedDescription using the singular or plural number may also include theplural or singular number respectively. The word “or” in reference to alist of two or more items, covers all of the following interpretationsof the word: any one of the items in the list, all of the items in thelist, and any combination of the items in the list. Likewise the term“and/or” in reference to a list of two or more items, covers all of thefollowing interpretations of the word: any one of the items in the list,all of the items in the list, and any combination of the items in thelist.

Depending on the embodiment, certain operations, acts, events, orfunctions of any of the algorithms described herein can be performed ina different sequence, can be added, merged, or left out altogether(non-limiting example: not all are necessary for the practice of thealgorithms). Moreover, in certain embodiments, operations, acts,functions, or events can be performed concurrently, non-limitingexamples: through multi-threaded processing, interrupt processing, ormultiple processors or processor cores or on other parallelarchitectures, rather than sequentially.

The various illustrative logical blocks, modules, routines, andalgorithm steps described in connection with the embodiments disclosedherein can be implemented as electronic hardware, or as a combination ofelectronic hardware and executable software. To clearly illustrate thisinterchangeability, various illustrative components, blocks, modules,and steps have been described above generally in terms of theirfunctionality. Whether such functionality is implemented as hardware, oras software that runs on hardware, depends upon the particularapplication and design constraints imposed on the overall system 100.The described functionality can be implemented in varying ways for eachparticular application, but such implementation decisions should not beinterpreted as causing a departure from the scope of the disclosure.

Moreover, the various illustrative logical blocks and modules describedin connection with the embodiments disclosed herein can be implementedor performed by a machine, such as a processor device, a digital signalprocessor (DSP), an application specific integrated circuit (ASIC), afield programmable gate array (FPGA) or other programmable logic device,discrete gate or transistor logic, discrete hardware components, or anycombination thereof designed to perform the functions described herein.A processor device can be a microprocessor, but in the alternative, theprocessor device can be a controller, microcontroller, or combinationsof the same, or the like. A processor device can include electricalcircuitry configured to process computer-executable instructions. Inanother embodiment, a processor device includes an FPGA or otherprogrammable device that performs logic operations without processingcomputer-executable instructions. A processor device can also beimplemented as a combination of computing devices, non-limitingexamples: a combination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration. Although described hereinprimarily with respect to digital technology, a processor device mayalso include primarily analog components. For example, some or all ofthe signal processing algorithms described herein may be implemented inanalog circuitry or mixed analog and digital circuitry. A computingenvironment can include any type of computer system, including, but notlimited to, a computer system based on a microprocessor, a mainframecomputer, a digital signal processor, a portable computing device, adevice controller, or a computational engine within an appliance, toname a few.

The elements of a method, process, routine, or algorithm described inconnection with the embodiments disclosed herein can be embodieddirectly in hardware, in a software module executed by a processordevice, or in a combination of the two. A software module can reside inRAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory,registers, hard disk, a removable disk, a CD-ROM, or any other form of anon-transitory computer-readable storage medium. An exemplary storagemedium can be coupled to the processor device such that the processordevice can read information from, and write information to, the storagemedium. In the alternative, the storage medium can be integral to theprocessor device. The processor device and the storage medium can residein an ASIC. The ASIC can reside in a user terminal. In the alternative,the processor device and the storage medium can reside as discretecomponents in a user terminal.

Further, the processing of the various components of the illustratedsystems can be distributed across multiple machines, networks, and othercomputing resources. In addition, two or more components of a system canbe combined into fewer components. Various components of the illustratedsystems can be implemented in one or more virtual machines, rather thanin dedicated computer hardware systems and/or computing devices.

Virtualization technologies allow a single physical computing device tohost one or more instances of a virtual machine, which virtual machineinstance appears to a user as an independent computing device. Withvirtualization, the host computing device can create, maintain, delete,or otherwise manage virtual machines instances in a dynamic manner. Inturn, users can request computing resources, including single computingdevices or a configuration of networked computing devices, and beprovided with virtual machine instances that provide the requestedcomputing resources.

An instance of a virtual machine may be configured to provide specificfunctionality. For example, a virtual machine instance may be associatedwith different combinations of software applications and operatingsystems or operating system configurations to enable a virtual machineto provide different desired functionalities, or to provide similarfunctionalities more efficiently.

Any patents and applications and other references noted above, includingany that may be listed in accompanying filing papers, are incorporatedherein by reference. Aspects of the invention can be modified, ifnecessary, to employ the systems, functions, and concepts of the variousreferences described above to provide yet further implementations of theinvention.

These and other changes can be made to the invention in light of theabove Detailed Description. While the above description describescertain examples of the invention, and describes the best modecontemplated, no matter how detailed the above appears in text, theinvention can be practiced in many ways. Details of the system may varyconsiderably in its specific implementation, while still beingencompassed by the invention disclosed herein. As noted above,particular terminology used when describing certain features or aspectsof the invention should not be taken to imply that the terminology isbeing redefined herein to be restricted to any specific characteristics,features, or aspects of the invention with which that terminology isassociated. In general, the terms used in the following claims shouldnot be construed to limit the invention to the specific examplesdisclosed in the specification, unless the above Detailed Descriptionsection explicitly defines such terms. Accordingly, the actual scope ofthe invention encompasses not only the disclosed examples, but also allequivalent ways of practicing or implementing the invention under theclaims.

To reduce the number of claims, certain aspects of the invention arepresented below in certain claim forms, but the applicant contemplatesthe various aspects of the invention in any number of claim forms. Forexample, while only one aspect of the invention may be recited as ameans-plus-function claim under 35 U.S.C sec. 112(f) (AIA), otheraspects may likewise be embodied as a means-plus-function claim, or inother forms, such as being embodied in a computer-readable medium. Anyclaims intended to be treated under 35 U.S.C. § 112(f) will begin withthe words “means for”, but use of the term “for” in any other context isnot intended to invoke treatment under 35 U.S.C. § 112(f). Accordingly,the applicant reserves the right to pursue additional claims afterfiling this application, in either this application or in a continuingapplication.

Disjunctive language such as the phrase “at least one of X, Y, or Z,”unless specifically stated otherwise, is otherwise understood with thecontext as used in general to present that an item, term, etc., may beeither X, Y, or Z, or any combination thereof (non-limiting examples: X,Y, and/or Z). Thus, such disjunctive language is not generally intendedto, and should not, imply that certain embodiments require at least oneof X, at least one of Y, or at least one of Z to each be present.

Unless otherwise explicitly stated, articles such a “a” or “an” shouldgenerally be interpreted to include one or more described items.Accordingly, phrases such as “a device configured to” are intended toinclude one or more recited devices. Such one or more recited devicescan also be collectively configured to carry out the stated recitations.For example, “a processor configured to carry out recitations A, B andC” can include a first processor configured to carry out recitation Aworking in conjunction with a second processor configured to carry outrecitations B and C.

While the above detailed description has shown, described, and pointedout novel features as applied to various embodiments, it can beunderstood that various omissions, substitutions, and changes in theform and details of the devices or algorithms illustrated can be madewithout departing from the spirit of the disclosure. As can berecognized, certain embodiments described herein can be embodied withina form that does not provide all of the features and benefits set forthherein, as some features can be used or practiced separately fromothers. The scope of certain embodiments disclosed herein is indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

The previous description of the disclosed implementations is provided toenable a person skilled in the art to make or use the present invention.Various modifications to these implementations will be readily apparentto those skilled in the art, and the generic principles defined hereincan be applied to other implementations without departing from thespirit or scope of the invention. Furthermore, although described abovewith reference to medical devices and procedures, it will be understoodthat the embodiments described herein can be applied to other systems inwhich non-medical objects are tracked and non-medical image streams arereceived, and virtual representations are displayed on a display and/orsystems in which multiple objects are displayed on a display within avirtual space, such as within a virtual 3D space. Thus, the presentinvention is not intended to be limited to the implementations shownherein but is to be accorded the widest scope consistent with theprinciples and novel features disclosed herein.

1-20. (canceled)
 21. A computer system in communication with a display,the computer system comprising a computer processor and a non-transitorystorage medium, wherein the computer system is configured to: determinean emplacement of a medical display object; determine an emplacement ofa plane-of-interest; determine an emplacement of a ghost medical displayobject based at least in part on the determined emplacement of themedical display object and the determined emplacement of theplane-of-interest; and causing the display to concurrently display, in avirtual 3D space: at least a portion of the medical display object, andat least a portion of the ghost medical display object on theplane-of-interest.
 22. The system of claim 21, wherein the computersystem is further configured to: receive first emplacement dataassociated with a first medical device, wherein to determine theemplacement of the medical display object, the computer system isconfigured to determine the emplacement of the medical display objectbased at least in part on the received first emplacement data; andreceive second emplacement data associated with a second medical device,wherein to determine the emplacement of the plane-of-interest, thecomputer system is configured to determine the emplacement of theplane-of-interest based at least in part on the received secondemplacement data.
 23. The system of claim 21, wherein the computersystem is further configured to: receive a medical image correspondingto the second medical device; and determine an emplacement of themedical image based at least in part on the received second emplacementdata, wherein to determine the emplacement of the plane-of-interest, thecomputer system is configured to determine the emplacement of theplane-of-interest based at least in part on the determined emplacementof the medical image.
 24. The system of claim 21, wherein to determinethe emplacement of the ghost medical display object, the computer systemis further configured to, at least one of: determine at least one ofportions, points, or coordinates of the plane-of-interest that areclosest to the medical display object, determine an arc pathcorresponding to the medical display object and the plane-of-interest,determine which of a plurality of connecting lines intersect the medicaldisplay object, wherein the plurality of connecting lines extendorthogonally from the plane-of-interest to the medical display object,or use a virtual light source that is at least as large as the medicaldisplay object to be projected, wherein the virtual light source isdistal to the plane-of-interest relative to the medical display object.25. The system of claim 21, wherein the ghost medical display objectcomprises at least one of: a virtual medical device that corresponds toa physical medical device, wherein the physical medical device comprisesa joint member having a joint member axis, a first limb member having afirst axis, and a second limb member having a second axis that isdifferent from the joint member axis and the first axis; a medical imageslice that corresponds to a real-time medical image of a medical imagingdevice; or an affected region of the virtual medical device, wherein theaffected region corresponds to a volume located between the first limbmember and the second limb member.
 26. The system of claim 21, whereinthe plane-of-interest corresponds to at least one of a plane of amedical device or an image plane.
 27. The system of claim 21, whereinthe medical display object corresponds to an affected region of amedical device, wherein the medical device comprises a joint memberhaving a joint member axis, a first limb member having a first axis, anda second limb member having a second axis that is different from thejoint member axis and the first axis, and wherein the affected regioncorresponds to at least one of: a volume located between the first limbmember and the second limb member, a stapling path of at least onestaple of the medical device, or a cutting path of a knife of themedical device.
 28. The system of claim 21, wherein the medical displayobject corresponds to at least one of a transecting stapler, a grasper,or a vessel sealer, and wherein the plane-of-interest corresponds to animaging plane of a medical imaging device.
 29. A computer system incommunication with a display, the computer system comprising a computerprocessor and a non-transitory storage medium, wherein the computersystem is configured to: determine an emplacement of a virtual medicaldevice corresponding to a physical medical device, the physical medicaldevice comprising a joint member having a joint member axis, a firstlimb member having a first axis, and a second limb member having asecond axis that is different from the joint member axis and the firstaxis; determine an emplacement of an affected region based at least inpart on the determined emplacement of the virtual medical device; andcause the display to display, in a virtual 3D space, at least one of: atleast a portion of the affected region based at least in part on thedetermined emplacement of the virtual medical device, or an intersectionindicator indicating an intersection between an image plane and atrajectory of the affected region along a drive path that is parallel tothe joint member axis.
 30. The system of claim 29, wherein the computersystem is further configured to: receive emplacement data associatedwith the physical medical device, wherein to determine the emplacementof the virtual medical device, the computer system is configured todetermine the emplacement of the virtual medical device based at leastin part on the received emplacement data; and receive operatingparameters corresponding to the physical medical device, wherein todetermine the emplacement of the affected region, the computer system isconfigured to determine emplacement of the affected region is furtherbased at least in part on the received operating parameters.
 31. Thesystem of claim 29, wherein the computer system is further configured tocause the display to further display, in the virtual 3D space, at leasta portion of the virtual medical device based at least in part on thedetermined emplacement of the virtual medical device.
 32. The system ofclaim 29, wherein the affected region corresponds to a volume locatedbetween the first limb member and the second limb member.
 33. The systemof claim 29, wherein the physical medical device comprises a medicalstapler, wherein the affected region comprises a stapling path of themedical stapler.
 34. The system of claim 29, wherein the affected regioncomprises a cutting path of a knife of the physical medical device. 35.The system of claim 29, wherein the medical device comprises at leastone of a transecting stapler, a grasper, or a vessel sealer.
 36. Acomputer system in communication with a display, the computer systemcomprising a computer processor and a non-transitory storage medium,wherein the computer system is configured to: determine an emplacementof a virtual medical device; determine an emplacement of a medical imageslice; determine a first intersection between an image planecorresponding to the medical image slice and a trajectory of a firstportion of the virtual medical device along a first trajectory path ofthe virtual medical device; determine a second intersection between theimage plane and a trajectory of a second portion of the virtual medicaldevice along the first trajectory path of the virtual medical device;and cause the display to concurrently display, in a virtual 3D space: atleast a portion of the medical image slice based at least in part on thedetermined emplacement of the medical image slice, a first intersectionindicator corresponding to the first intersection, and a secondintersection indicator corresponding to the second intersection.
 37. Thesystem of claim 36, wherein the virtual medical device corresponds to aphysical medical device, the physical medical device comprising a jointmember having a joint member axis, a first limb member having a firstaxis, and a second limb member having a second axis that is differentfrom the joint member axis and the first axis, wherein the first portionof the virtual medical device corresponds to the first limb member,wherein the second portion of the virtual medical device corresponds tothe second limb member.
 38. The system of claim 37, wherein the medicaldevice comprises at least one of a transecting stapler, a grasper, or avessel sealer.
 39. The system of claim 37, wherein the first trajectorypath is parallel to at least one of the joint member axis, the firstaxis, or the second axis.
 40. The system of claim 36, wherein thecomputer system is further configured to cause display to furtherconcurrently display, in the virtual 3D space, at least one of: a firsttrajectory indicator corresponding to the trajectory of the firstportion of the virtual medical device along the first trajectory path,or a second trajectory indicator corresponding to the trajectory of thesecond portion of the virtual medical device along the first trajectorypath.